Modeling Second-Language Learning from a Psychological Perspective

Alexander S. Rich Pamela J. Osborn Popp David J. Halpern

Anselm Rothe Todd M. Gureckis

Department of Psychology, New York University

asr443, pamop, david.halpern, anselm, todd.gureckis@nyu.edu

Abstract

Psychological research on learning and memory has tended to emphasize small-scale laboratory studies. However, large datasets of people using educational software provide opportunities to explore these issues from a new perspective. In this paper we describe our approach to the Duolingo Second Language Acquisition Modeling (SLAM) competition which was run in early 2018. We used a well-known class of algorithms (gradient boosted decision trees), with features partially informed by theories from the psychological literature. After detailing our modeling approach and a number of supplementary simulations, we reflect on the degree to which psychological theory aided the model, and the potential for cognitive science and predictive modeling competitions to gain from each other.

1 Introduction

This paper reports the approach used by our team, which finished in third place for the Spanish-English (es), second place for English-Spanish (en), and third place for English-French (fr) datasets.

For our approach to the SLAM competition, we decided to use a generic and fairly flexible model structure that we provided with hand-coded, psychologically inspired features. We therefore positioned our entry to SLAM somewhat in between the approaches mentioned above. Specifically, we used gradient boosting decision trees (GBDT, Ke et al., 2017) for the model structure, which is a powerful classification algorithm that is known to perform well across various kinds of datasets. Like deep learning, GBDT can extract complex interactions among features, but it has some advantages including faster training and easier integration of diverse inputs.

We then created a number of new psychologically-grounded features for the SLAM dataset covering aspects such as user perseverance, learning processes, contextual factors, and cognate similarity. After finding a model that provided the best held-out performance on the test dataset, we conducted a number of "lesioning" studies where we selectively removed features from the model and re-estimated the parameters in order to assess the contribution of particular types of features. We begin by describing our overall modeling approach, and then discuss some of the lessons learned from our analysis.

2 Task Approach

We approached the task as a binary classification problem over instances. Each instance was a single word within a sentence of a translation exercise and the classification problem was to predict whether a user would translate the word correctly or not. Our approach can be divided into two components—constructing a set of features that is informative about whether a user will answer an instance correctly.

2.1 Feature Engineering

We used a variety of features, including features directly present in the training data, features constructed using the training data, and features that use information external to the training data. Except where otherwise specified, categorical variables were one-hot encoded.

2.1.1 Exercise features

We encoded the exercise number, client, session format, and duration (i.e., number of seconds to complete the exercise), as well as the time since the user started using Duolingo for the first time.

2.1.2 Word features

Using spaCy, we lemmatized each word to produce a root word. Both the root word token and the original token were used as categorical features. Due to their high cardinality, these features were not one-hot encoded but were preserved in single columns and handled in this form by the model.

Along with the tokens themselves we encoded each instance word's part of speech, morphological features, and dependency edge label. We noticed that some words in the original dataset were paired with the wrong morphological features, particularly near where punctuation had been removed from the sentence. To fix this, we reprocessed the data using Google SyntaxNet.

We also encoded word length and several word characteristics gleaned from external data sources. Research in psychology has suggested certain word features that play a role in how difficult a word is to process. Two such features that have somewhat independent effects are word frequency and age-of-acquisition. We included a feature that encoded the frequency of each word in the language being acquired, calculated from external sources.

2.1.3 User features

We calculated several user-level features related to the "learning type" of a user. In particular, we encoded features related to psychological constructs such as motivation and diligence of a user.

We speculated that more motivated users would complete more exercises. To capture this, we grouped each user's exercises into "bursts." We used three concrete features about these bursts, namely the mean and median number of exercises within bursts as well as the total number of bursts of a given user.

As a proxy for diligence, we estimated that a diligent user might use the app regularly at the same time of day. Instead, we were able to extract for each exercise the time of day relative to the first time a user had used the app.

2.1.4 Positional features

To account for the effects of surrounding words on the difficulty of an instance, we created features related to the instance word's context in the exercise. These included the token of the previous word, the next word, and the instance word's root in the dependency tree, all stored in single columns as with the instance token itself.

2.1.5 Temporal features

A user's probability of succeeding on an instance is likely related to their prior experience with that instance. We encoded the number of times the current exercise's exact sentence had been seen before by the user.

We also encoded a set of features recording past experience with the particular instance word. For each token, we tracked user performance through four weighted error averages. Each error term starts at zero and is updated with each encounter with an instance of the token.

2.2 Modeling

After generating all of the features for the training data, we trained GBDT models to minimize log loss. GBDT works by iteratively building regression trees, each of which seeks to minimize the residual loss from prior trees.

To tune model hyper-parameters and evaluate the usefulness of features, we first trained the models on the train data set and evaluated them on the dev data set. Once the model structure was finalized, we trained on the combined train and dev data and produced predictions for the test data.

2.3 Performance

For reference, we achieved various performance metrics for the datasets. For example, our performance on the Spanish-English dataset placed us close to the Duolingo-provided baseline model.

Table 1: Parameters of final LightGBM models.

3 Feature Removal Experiments

To understand which features or groups of features were most important to our model's predictions, we conducted a set of experiments where we removed groups of features and re-trained the model, evaluating performance on the dev set. The results of the lesion experiments reveal insights into the contribution of various features.

4 Discussion

When approaching the Duolingo SLAM task, we hoped to leverage psychological insights in building our model. While some features, such as age-of-acquisition, were helpful, the model gained its power more from applying a powerful inference technique to raw input data than from hand-crafted features.

In sum, we believe the Duolingo SLAM dataset provides interesting opportunities for cognitive science and psychology.