CLUF: a Neural Model for Second Language Acquisition Modeling

Shuyao Xu Singsound Inc. Beijing, China xushuy@singsound.com

Jin Chen Singsound Inc. Beijing, China chenjin@singsound.com

Long Qin Singsound Inc. Beijing, China qinlong@singsound.com

Abstract

Second Language Acquisition Modeling is the task to predict whether a second language learner would respond correctly in future exercises based on their learning history. In this paper, we propose a neural network based system to utilize rich contextual, linguistic and user information. Our neural model consists of a Context encoder, a Linguistic feature encoder, a User information encoder and a Format information encoder (CLUF). Furthermore, a decoder is introduced to combine such encoded features and make final predictions. Our system ranked in first place in the English track and second place in the Spanish and French track with an AUROC score of 0.861, 0.835 and 0.854 respectively.

1 Introduction

Education systems that can adapt to the presenting of educational materials according to students' personal learning needs have great potential. Specifically, in the area of second language learning, we try to predict whether the learning materials are too easy or too hard for language learners. Therefore, we study the Second Language Acquisition Modeling (SLAM) task to build a model of the language learning process.

Bayesian Knowledge Tracing (BKT) (Corbett and Anderson, 1994; Pardos and Heffernan, 2010; Pelanek, 2017) that models students' knowledge over time is a well-established problem. It takes a Hidden Markov Model (HMM) with binary hidden states to represent knowledge acquisition for each concept separately. BKT had been successfully applied to subjects like mathematics and programming, where a limited number of concepts can be predefined. However, in language learning, it's difficult to define a small number of concepts especially when the vocabulary size increases over time. Deep Knowledge Tracing (DKT) (Piech et al., 2015; Wilson et al., 2016) is a recent implementation of knowledge tracing which uses Recurrent Neural Networks (RNNs) to model student's learning trace. Although RNNs and its commonly used variants, such as Gated Recurrent Units (Cho et al., 2014) and Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), are capable of exploring dynamic temporal behavior for a time sequence, it's hard to model extremely long learning history that can range over months even years. Half-life Regression (Settles and Meeder, 2016) is a novel approach for the SLAM task, which combines a psycholinguistic model of human memory with modern machine learning techniques. It had demonstrated state-of-art performance for predicting student recall rates.

Mapping symbols, such as characters or words, into a continuous space is a popular method in natural language processing (Hinton, 1986; Mikolov et al., 2013; Pennington et al., 2014; Mikolov et al., 2017). It achieved remarkable success in many tasks, for example, neural language modeling (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2010), machine translation (Sutskever et al., 2014; Bahdanau et al., 2015), text classification (Lai et al., 2015; Zhang et al., 2015; Conneau et al., 2017), sentiment analysis (dos Santos and Gatti, 2014; Poria et al., 2015) and machine reading comprehension (Xiong et al., 2017; Hu et al., 2017). In this work, we introduce a similar neural approach for the SLAM task, where we use neural encoders to extract features from each exercise as well as metadata about student and session. To be specific, we build a Context encoder, a Linguistic feature encoder, a User information encoder and a Format information encoder (CLUF) to calculate high-level representations from characters, words, part-of-speech (POS) labels, syntactic dependency labels, user id and country, exercise type, client, etc.

Table 1 : The SLAM dataset statistics

2 Dataset

The Duolingo SLAM dataset (Settles et al.,2018)

es: English learners (who already speak Spanish)

en: Spanish learners (who already speak English)

en: French learners (who already speak English)

es has the lowest positive ratio, while es en has the highest out-of-vocabulary (OOV) ratio.

Table 2 shows the features provided with the SLAM dataset. In our system, we used all features except the morphology features and syntactic dependency edges, as we did not get any improvement during experiments. Perhaps it is because that the neural networks already encoded similar information from characters, words and their syntactic dependency labels.

3 Method

We used in total four encoders to model the students' learning behavior. Inputs to these encoders are embeddings learned from one-hot representations of raw features. The context encoder consists of a character level LSTM encoder and a word level LSTM encoder. The linguistic feature encoder is also a LSTM model, where POS and syntactic dependency embedding are concatenated to-

Table2 shows the features provided with the SLAM dataset.

3.1 Context Encoder

The context encoder operates at both the word level and the character level. The word level encoding is capable of capturing better semantics and longer dependency than the character level encoding. But learning new words is a key part in language learning. By modeling the character sequence, we may be able to learn certain word formation rules, therefore partially avoid the OOV problem.

The word level context encoder is a Bidirectional LSTM model. Given a sequence of words represented as one-hot vectors $ ( w_{1}, w_{2}, \dots, w_{N} ) $ we can get the word embedding of $ w_{t} $ as

$$ \ w_{1},:w_{2},:..., :w_{N}) $$ $$\ x_{t}=E_{w}\cdot w_{t},$$

where $ E_{w} $ is the word embedding matrix, which is learned during training.

$$ E_{w} $$ $$\ x_{t}. $$ Given the input vector $ x_{t} $, the forward, backward, and combined activations of the j-th hidden layer are computed as $$ \begin{aligned}{f_{t}^{j}}&{{}=L S T M(f_{t-1}^{j},:f_{t}^{j-1})}\ {b_{t}^{j}}&{{}=L S T M(b_{t+1}^{j},:b_{t}^{j-1})}\ {g_{t}}&{{}=[f_{t}^{K_{0}},:b_{t}^{K_{0}}],}\ \end{aligned} $$ where $ K_{0} $ is the number of layers of the network, j=1,2,..., $ K_{0}. $ $$ K_{0} $$ $$j=1,2,...,K_{0} $$

The character level context encoder is a hierarchical LSTM model. Given a sequence of one-hot representations of characters in word $ w_{t} $ $ (c_{1}, c_{2}, \dots, c_{M}) $, we can get the embedding of $ c_{i} $ as $$ \ w_{t}, $$ \big(c_{1},;c_{2},;..., :c_{M}\big) $$ $$h_{i}^{0}=E_{c}\cdot c_{i},$$

where $ E_{c} $ is the character embedding matrix which is learned during training. $$ E_{c} $$ The outputs of the lookup layer are then fed into a multilayer LSTM unit $$ \begin{array}{r}{h_{i}^{j}=\mathit{L S T M}(h_{i-1}^{j},:h_{i}^{j-1})}\ {H_{w_{t}}=(h_{1}^{K_{1}},:h_{2}^{K_{1}},:..., :h_{M}^{K_{1}}),}
\end{array} $$ where $ K_{1} $ is the number of layers of the LSTM, j=1,2,..., $ K_{1}. $ $$ j=1,2,...,K_{1} $$

Figure 2: The Character Level Context Encoder

The mean-over-time (MoT) layer takes $ H_{w_{t} } $ as inputs $$ H_{w_{t}} $$ h_{w_{t}}=\frac{1}{M}\sum_{i=1}^{M}h_{i}^{K_{1}}, $$ Then the outputs of the MoT layer ( $$(h_{w_{1}},h_{w_{2}},...$$ $$h_{w_{N} ) $$ $$\begin{aligned}{\hat{f}{t}^{j}}&{{}=L S T M(\hat{f}{t-1}^{j}, :\hat{f}{t}^{j-1})}\ {\hat{b}{t}^{j}}&{{}=L S T M(\hat{b}{t+1}^{j},:\hat{b}{t}^{j-1})}\ {\hat{g}{t}}&{{}=[\hat{f}{t}^{K_{2}},:\hat{b}{t}^{K{2}}],}\ \end{aligned} $$ where $ K_{2} $ is the number of layers of the BiLSTM, j=1,2,..., $ K_{2}. $ $$ K_{2} $$ j=1,2,...,K_{2} $$

The final outputs of the context encoder are computed as: $$ O=(big,o_{1},:o_{2},:..., :o_{N}\big), $$ where $ o_{t}=g_{t}+\hat{g}_{t}. $

$$ o_{t}=g_{t}+\hat{g}_{t}. $$

3.2 Linguistic Feature Encoder

The linguistic feature encoder is also a LSTM model. Similar to the context encoder, we trained embedding representations of the POS labels and the syntactic dependency labels. The POS embeddings and syntactic dependency embeddings are concatenated together and then fed into a LSTM unit, $$ \begin{aligned}{}&{{}l_{t}^{0}=[pos_{t},:dpp{t}]}\ {}&{{}l{t}^{j}=L S T M(l l_t{-1}^{j},:l_{t}^{j-1})}\ {}&{{}L=(l_{1}^{K_{3}},:l_{2}^{K_{3}},:..., :l_{N}^{K_{3}}),}
\end{aligned} $$ where pos_tis the POS embedding of word dep $$w_{t} $$ $$pos_{t} $$ $$de_{p}{t} $$

3.3 User Encoder

The user encoder is a one-layer fully-connected feedforward network. The encoder takes user metadata as inputs $$ \begin{aligned}{}&{{}\ \mu^{0}=[u,:s,:{ay}]}\ {}&{{}\mu^{1}=tanh(W_{\mu}\cdot\mu^{0}+:b_{\mu}),}
\end{aligned} $$ where u is the embedding of the user id, s is the embedding of the user's nationality and days is the time since the student started learning this language. $ W_{\mu}, $ $ b_{\mu} $ are trained network parameters. We used the tanh activation function for the user encoder.

$$ W_{\mu}, b_{\mu} $$

3.4 Format Encoder

Similar to the user encoder, the format encoder is also a one-layer fully-connected feedforward network. The inputs are format, session, client, and the response time, $$ f^{0}=[format, s, session, c, time] $$ where $ W_{f}, $ $ b_{f} $ are trainable parameters. $$ f^{1}=tanh\big(W_{f}\cdot f^{0}+,b_{f}\big), $$ $$b_{f},$$

3.5 Decoder

The decoder takes the outputs (O, L, $ \mu^{1}, f^{1} $ ) of the context encoder, linguistic encoder, user encoder and format encoder as inputs. The prediction for word $ w_{t} $ in the given sequence $ (w_{1}, w_{2},...,w_{N}) $ is computed as $$(O,L,\mu^{1},f^{1}) $$ ative and positive samples and $$w_{t} $$ $$(w_{1},:w_{2},:..., :w_{N}) $$ $$\nu=min\big(W_{\nu}\cdot[\mu^{1},:f^{1}]+:b_{\nu}\big) $$ $$\gamma_{t} = \sigma \left(W_{\gamma} \cdot \left[ l_{t}^{K_{3}}, o_{t} \right] + b_{\gamma}\right) $$ $$p_{t}= $$ $$\sigma\big(W_{p}\cdot\big(\nu\odot\gamma_{t}\big)+:b_{p}\big), $$ where $ W_{\nu}, b_{\nu}, W_{\gamma}, b_{\gamma}, W_{p}, $ and $ b_{p} $ are trainable parameters. For decoding, we used the sigmoid activation function $ \sigma $ . $$W_{\nu}, b_{\nu}, W_{\gamma}, b_{\gamma}, W_{p} $$

3.6 Training

The model is trained to minimize the following loss function $$ \begin{aligned}{Loss&=-\frac{1}{N}\sum_{t=1}^{N}(\alpha y_{t}\cdot log(p_{t});+)}\ {}&{(1-\alpha)(1-y_{t})\cdot log(1-p_{t})),}\end{aligned} $$

Table 3 : Results of the en es track.

Table 4 : Results of the es en track.

4 Experiments and Results

We considered the words that appear less than five times in the training data as unknown token. For students with more than one nationality, only the first one was used.

4.1 Experiments

) + \alpha)(1 − y_t) · log (1 − p_t)),

Table 5 : Results of the fr en track.

4.2 Results

Table 6 : The relative improvement over the base-

The evaluation metrics for the SLAM task were the Area Under the Receiver Operation Characteristic (AUROC) curve and the F1 score.

encoder was a two-layer LSTM. Both of the user encoder and format encoder were one-layer fully-connected feedforward networks.

es, es en, and fr en track, respectively. We ranked in first place in the en es track and second place in the es en and fr en track.

Table6 shows that CLUF gained significant imes and es en track were close, while the improvement on the en track was a bit lower. We think this is been (327k exercises) track has much es (824k exercises) en (732k exercises) track. and es

4.3 Discussion

Table 7 : Encoder analysis. LUF has no context encoder; CUF has no linguistic encoder; CLF has

over time; 4) the format encoder measures the difficulty level of different exercises on various clients.

line encoder was a two-layer LSTM. Both of the user encoder and format encoder were one-layer fully-

5 Conclusion

provements on all tracks compared to the baseline

Acknowledgments

fr cause the fr less training data than the en

Our intuition behind CLUF is to factorize raw features into four independent parts: 1) word surface guage model. Journal of machine learning research danau, and Yoshua Bengio. 2014. On the properties

, 3(Feb):1137–1155.