Neural sequence modelling for learner error prediction

Zheng Yuan
The ALTA Institute
Department of Computer Science and Technology
University of Cambridge
zheng.yuan@cl.cam.ac.uk

Abstract

This paper describes our use of two recurrent neural network sequence models: sequence labelling and sequence-to-sequence models, for the prediction of future learner errors in our submission to the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM). We show that these two models capture complementary information as combining them improves performance. Furthermore, the same network architecture and group of features can be used directly to build competitive prediction models in all three language tracks, demonstrating that our approach generalises well across languages.

1 Introduction

Most recent work on second language acquisition (SLA) has focused on intermediate-to-advanced learners in assessment settings driven by a series of shared tasks (Dale and Kilgarriff, 2011; Dale et al., 2012; Ng et al., 2013, 2014; Lee et al., 2015, 2016; Daudaravicius et al., 2016). The 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM) (Settles et al., 2018) targets early stage learners and aims to provide personalised learning instructions. Participating teams are provided with transcripts from exercises submitted by learners over their first 30 days of learning on Duolingo, which are annotated for token (word) level errors. The task is to predict what errors each learner will make in the future based on their learning history. There are three language tracks in this shared task:

• es: native Spanish speakers learning English;
• en: native English speakers learning Spanish;
• fr: native English speakers learning French.

Teams can either focus on a particular language track, or explore generalised models and features across all three languages.

Inspired by the success of neural sequence models in grammatical error detection and correction (Yuan and Briscoe, 2016; Rei and Yannakoudakis, 2016; Yannakoudakis et al., 2017; Schmaltz et al., 2017), we propose two recurrent neural network sequence models for this problem: sequence labelling and sequence-to-sequence modelling. We demonstrate the utility of these two models for the future learner error prediction task. We also provide evidence of performance gains by using an ensemble of these two models, suggesting that they are complementary to each other.

2 Approach

2.1 Neural sequence labelling

We treat error prediction as a sequence labelling problem. Similar to Yannakoudakis et al. (2017), we construct a bidirectional recurrent neural network for detecting future learner errors. Unlike their system, error-free and correct sequences are fed into our model, and the goal is to predict where a learner is likely to make token-level errors based on their learning history. The model receives a sequence of tokens $ \mathbf{x}=\left(x_{1},x_{2},\dots,x_{T}\right) $ as input, and assigns a label y to each input token x. A bidirectional long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) is used to learn context-specific representations:

$$ \mathtt{x}=(x_{1},x_{2},...,x_{T}) $$

$$ \overrightarrow{h_{t}}=\operatorname{L S T M}(x_{t},\overrightarrow{h_{t-1}}) $$

(1)

$$ \overleftarrow{h_{t}}=\operatorname{L S T M}(x_{t},\overleftarrow{h_{t+1}}) $$

(2)

$$ h_{t}=[\overrightarrow{h_{t}};\overleftarrow{h_{t}}] $$

(3)

where $ \overrightarrow{h_{t}} $ is the hidden state of the forward-moving LSTM at time t, that reads the input sequence from the first token to the last; $ \overleftarrow{h_{t}} $ is the hidden state of the backward-moving LSTM at time t, which reads the input sequence in reverse order; and $ h_{t} $ is the concatenation of both hidden states, that captures both historical and future sequential information.

A softmax output layer predicts the label distribution for each input token, given the whole input sequence x:

$$ p(y_{t}\big|\mathtt{x})=\mathtt{s o f t m a x}(W_{o}h_{t}) $$

where $ W_{o} $ is an output weight matrix.

We optimise the model by minimising categorical cross-entropy between the predicted label distributions and the gold labels:

$$ E=-\sum_{t=1}^{T}\log p(y_{t}|\mathbf{x}) $$

2.2 Sequence-to-sequence modelling

We introduce another model for the task of future learner error prediction: a sequence-to-sequence model. The input sequence is represented in the same way:

$$ \mathbf{x} = \left(x_{1}, x_{2}, \dots, x_{T}\right) $$

The hidden states in this sequence are defined as:

$$ \mathbf{h}=(h_{1},h_{2},...,h_{T}) $$

The output sequence is denoted as:

$$\mathbf{y} ={y_{1},y_{2},...,y_{t-1}} $$

The prediction is made as follows:

$$ p(y_{t}|{y_{1},...,y_{t-1}},\mathtt{x})=\mathtt{s o f t m a x}(W_{o}s_{t}) $$

where $ W_{o} $ is a decoder output weight matrix, and $ s_{t} $ is the hidden state of the LSTM decoder at decoding time t:

$$ s_{t}=\mathrm{L S T M}(s_{t-1},y_{t-1},c_{t}) $$

where $ c_{t} $ is the input sequence representation for predicting the output token $ y_{t} $ , and is calculated using a soft attention mechanism:

$$ c_{t}=\sum_{j=1}^{T}(\alpha_{tj}h_{j}) $$

The weight $ \alpha_{tj} $ is computed with a softmax function:

$$ \alpha_{tj}=\frac{\exp(e_{tj})}{\sum_{k=1}^{T}\exp(e_{tk})} $$

A feedforward neural network is used to represent this calculation:

$$ e_{tj}=\operatorname{t a n h}\bigl(W_{\alpha}s_{t-1}+U_{\alpha}h_{j}\bigr) $$

3 Feature space

3.1 Exercise-level feature set

• type: the exercise session type (translate, tapreverse, or listen);
• client: the learner's device platform (android, ios, or web);
• country: the country from which the learner has done the exercise.

3.2 Token-level feature set

• part of speech (POS): the POS tag of the word;

3.3 Language-specific feature set

• dependency edge label (DEP): the grammatical relation (GR) between the word and its head.
• CEFR word level: The Common European Framework of Reference (CEFR) describes what language learners can do at different stages of their learning and defines language proficiency in six levels: A1, A2, B1, B2, C1, and C2.

4 Experiments and results

4.1 Dataset and evaluation

The shared task dataset comprises answers submitted by more than 6,000 Duolingo users over the course of their first 30 days. Token-level binary labels are provided:

The dataset is partitioned sequentially into training, development, and test sets, which all contain the same group of learners.

4.3 Results

Results for the sequence labelling models are presented in Table 1 and all our models outperform the baseline. We start by adding exercise-level features incrementally. Introducing new exercise-level features yields consistent improvements in overall performance.

5 Official evaluation results

Results for our submitted systems on the test sets for all three tracks:

6 Conclusions and future work

In this paper, we have described the use of recurrent neural sequence labelling and sequence-to-sequence models for future learner error prediction. Plans for future work include combining the training and development sets to train new models.