TMU System for SLAM-2018

Masahiro Kaneko
Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan

Abstract

We introduce the TMU systems for the second language acquisition modeling shared task (Settles et al., 2018). To model learner error patterns, it is necessary to maintain a considerable amount of information regarding the type of exercises learners have been.

1 Introduction

The field of psychology, linguistics, and pedagogy as the second language acquisition modeling shared task is an interesting research topic in the fields of psychology, linguistics, and pedagogy (Piech et al., 2015; Khajah et al., 2014, 2016).

2 2018 Duolingo Shared Task on SLAM

We used data from Duolingo in this shared task. translate: where learners translate written prompt from the tap item, where learners construct an answer given a set of words modeling.

• English learners (who already speak Spanish)
• Spanish learners (who already speak English)
• French learners (who already speak English)

3 TMU System

To track a lot of learner’s histories, our proposed system has two components: (1) a Prediction Bi-LSTM and (2) a History LSTM. It is expected to track a huge history of the learned exercises.

In prediction, we receive exercise as input and make predictions on word-level. Using Bi-LSTM for sequence labeling on exercise level, e.g., information as POS tags or dependency edge labels allows us to share information within each exercise for better prediction. We perform training by feeding input exercises arranged in chronological order for each learner.

3.1 Features

We use features (1-7) included in the dataset and labels for language identification. The TMU system has two components: an additional component that predicts whether a learner has made an error regarding the learned exercises and the words that they have input.

3.2 Prediction Bidirectional LSTM

We used bidirectional LSTM (Bi-LSTM) to predict word-level exercises based on inputs from the features above.

3.3 History LSTM

To correctly predict each exercise, the inputs to the LSTM systems must be arranged in chronological order to provide relevant context from past exercises.

3.4 Training

The objective function is defined as follows:
[ L_{\theta} = \frac{1}{|D|} \sum_{(x,y)\in D} \operatorname{log}p(y|x;\theta) ]
where D is the dataset.

3.5 Testing

During our test, predictions were made on exercises of the test data arranged in chronological order. In addition, we performed ensemble predictions. The parameters of ensemble models are initialized with different values. As the final prediction result, we used the average of the probability outputs of each Prediction Bi-LSTM. Each system used its own converted probability outputs of the Prediction Bi-LSTM component as gold answers.

4 Experiments

4.1 Experiment Settings

4.2 Results

Table 4 lists the results of SLAM for English learners, Spanish learners.

4.3 Analysis of Tracking History

5 Conclusion

In this study, we described the TMU system for the 2018 SLAM Shared Task. Our system is based on using Bi-LSTM for predicting word-level exercises and integrating history tracking to improve prediction accuracy. Key findings suggest that utilizing historical data enhances performance.