Second Language Acquisition Modeling

Burr Settles

∗ Chris Brust ∗ Erin Gustafson ∗ Masato Hagiwara ∗ Nitin Madnani † ∗ Duolingo, Pittsburgh, PA, USA † ETS, Princeton, NJ, USA {burr,chrisb,erin,masato}@duolingo.com nmadnani@ets.org

Abstract Then we report on the results of a “shared task” challenge organized by the authors using this SLA We present the task of second language acqui- sition (SLA) modeling. Given a history of errors made by learners of a second language, the task is to predict errors that they are likely to make at arbitrary points in the future. We describe a large corpus of more than 7M words produced by more than 6k learners of English, Spanish, and French using Duolingo, a popular online language-learning app. Then we report on the results of a shared task challenge aimed at studying the SLA task via this corpus, which attracted 15 teams and synthesized work from various fields including cognitive science, linguistics, and machine learning.

1 Introduction

As computer-based educational apps increase in popularity, they generate vast amounts of student learning data which can be harnessed to drive personalized instruction. While there have been some recent advances for educational software in domains like mathematics, learning a language is more nuanced, involving the interaction of lexical knowledge, morpho-syntactic processing, and several other skills. Furthermore, most work that has applied natural language processing to language learner data has focused on intermediate-to-advanced students of English, particularly in assessment settings. Much less work has been devoted to beginners, learners of languages other than English, or ongoing study over time. To create the SLA modeling corpus, we sampled from Duolingo users who registered for a course and reached at least the tenth row of skill icons within the month of November 2015. By limiting the data to new users who reach this level of course, we hope to better capture beginners’ broader language-learning process, including repeated interaction with vocabulary and grammar during their first 30 days of learning with Duolingo.

2 Shared Task Description

2.1 Corpus Collection

Our learner trace data comes from Duolingo: a free, award-winning, online language-learning platform. Since launching in 2012, more than 200 million learners worldwide have enrolled in Duolingo’s game-like courses, either via the website or mobile apps. Figure 1(a) is a screen-shot of the home screen, which specifies the game-like curriculum. Each icon represents a skill, aimed at teaching thematically or grammatically grouped words or concepts. Learners can tap an icon to access lessons of new material or to review material once all lessons are completed.

2.2 ThrHH Language Tracks

An important question for SLA modeling is the scope and effectiveness of language tracks offered by Duolingo. Spanish and French are the second and third most popular courses.

2.3 Label Prediction Task

The goal of the task is as follows. Given a history of token-level errors made by the learner in response to exercises, our aim is to predict future errors based on past behavior.

2.4 Data Set Format

Sample data from the resulting corpus can be found in Table 3 below, which outlines the distinct formats of data collected.

2.5 Challenge Timeline

The data were released in two phases to allow teams to compare different system parameters or features, focusing on the important contributions of each submitted model.

2.6 Evaluation

We use area under the ROC curve (AUC) as a common measure of ranking quality in classification tasks, interpreting it as the probability that the system will rank a randomly-chosen error above a randomly-chosen non-error.

3 Results

4 Related Work

SLA modeling is a rich problem, and presents a unique opportunity to synthesize work from various disciplines.

5 Meta-Analyses

In this section, we analyze the various modeling approaches and choices utilized by the participating teams.

6 Conclusion and Future Work

Among our key findings is the observation that, for this particular formulation of the task, the choice of learning algorithm appears to be more important than clever feature engineering. In particular, the most effective teams employed sequence models (e.g., RNNs) that can capture user performance over time, and tree ensembles (e.g., GBDTs) that can capture non-linear relationships among features. Furthermore, using a multitask framework—in this case, a unified model that leverages data from all three language tracks—can provide further improvements.