Predicting Second Language Learner Successes and Mistakes

by Means of Conjunctive Features

Yves Bestgen

Centre for English Corpus Linguistics
Université catholique de Louvain
Place Cardinal Mercier, 10 1348 Louvain-la-Neuve
yves.bestgen@uclouvain.be

Abstract

This paper describes the system developed by the Centre for English Corpus Linguistics for the 2018 Duolingo SLAM challenge. It aimed at predicting the successes and mistakes of second language learners on each of the words that compose the exercises they answered. Its main characteristic is to include conjunctive features, built by combining word ngrams with metadata about the user and the exercise. It achieved a relatively good performance, ranking fifth out of 15 systems. Complementary analyses carried out to gauge the contribution of the different sets of features to the performance confirmed the usefulness of the conjunctive features for the SLAM task.

1 Introduction

To develop the model, the organizers of the task is thus very different from the one at the root of many applications of natural language processing in the field of education that aim to automatically evaluate texts produced by second language learners. The traditional approach for the latter, which relies on linguistic indices more or less strongly correlated with text quality such as lexical richness, syntactic complexity and especially the presence of errors of different types, is obviously not applicable to the SLAM challenge.

Compared to the automatic evaluation of learner texts, the SLAM task has several advantages (+), but also several disadvantages (-):

• Each learner produced a relatively large number of responses allowing to estimate his or her level of competence;

• The learners' responses are spaced out in time making possible to try to model the evolution of their competence throughout their learning;

• The same exercises were presented to a large number of different learners making it possible to get a relatively good estimate of the difficulty of each of them;

• And above all, as indicated above, the prompt to be processed by the learner is provided, but not the actual answer.

As previous research of the CECL in this field deals with the question of automatic evaluation and only partially took into account the temporal dimension of learning, I chose to break down the problem in two sequences:

• Try to get the best prediction without using the sequential information available in the dataset.
• Add the sequential information and see whether it can improve the prediction.

Having not been successful in the second step, I focused this report on the first. It is therefore not really an attempt to model second language acquisition, but to predict the successes and mistakes of second language learners. The proposed system can be seen as a baseline system since it does not take into account the richest information made available.

The developed system achieved a relatively good performance since it ranks fifth out of 15 systems, but nevertheless at a respectable distance from the best systems. Its main characteristic is to include conjunctive features, built by combining several primitive features. In machine learning, these conjunctive features are classically obtained by means of a polynomial kernel, but this has the effect of greatly lengthening the time needed to learn the model. It was more efficient to obtain them manually and to use a (much faster) linear approach to learn the model.

The remainder of this report describes the datasets made available for this challenge, the system developed and the results obtained as well as the analyzes performed to get a better idea of the usefulness of the various components of the system.

2 Data

As explained in Settles et al. (2018), each instance to be categorized corresponded to a token of an exercise that has been presented to a user in one of three possible types of exercise, in one of three possible types of session and at a given time of his or her participation in the learning activities of the Duolingo platform. Several other metadata were provided for each exercise such as the country from which a user had done it. For each token, a series of morpho-syntactic features were also provided. The datasets were very large.

These datasets were divided by the organizers based on the TRAIN set to allow the fastest processing.

3 System

3.1 Main Features Used

Each instance (i.e., a token in an exercise) was encoded as a vector of 47 binary features, consisting of the following three feature sets:

• The token itself;
• The bigram including the next token;
• The bigram including the token that precedes it;
• The trigram including the two following tokens (if applicable).

The second set of features consisted of several other combinations, which increased the total feature set to a substantial number.

3.2 Sequential Information Use

All the features, which included a target token and had been previously seen by a user, were duplicated with a new value that reflected the number of times it had been seen, the proportion of mistakes this user made on it, and the time that had elapsed since he or she had seen it for the last time. These values were transformed by means of an exponential function. More details are not given on these features because they were very inefficient as shown in the analyzes reported below.

3.3 Procedure to Build the Models

The predictive models used during the development phase were built on logistic regression (L1-LR) available in the LIBLINEAR package. A series of tests led to setting the regularization parameter C.

4 Analyses and Results

In the tables presented below, T stands for the Token ngrams, M for the Metadata, U for User, F for Format, and S for Session. The performance metrics demonstrate the effectiveness of various feature sets in the predictive task.

4.2 In-depth Analysis of the Feature Sets

The conjunctive features, including those built from the metadata alone, made a significant contribution to performance. The model that only includes the token ngrams clearly underperformed. The metadata are necessary to achieve acceptable performance.

A second analysis was conducted to evaluate the impact of the three lengths of ngrams in the base model. The results indicated that the trigrams were not very useful contrary to the bigrams.

Acknowledgments

The author wishes to thank the organizers of the 2018 SLAM challenge for putting together this valuable event and the reviewers for their very constructive comments. This work was supported by the Fonds de la Recherche Scientifique - FNRS (grant number J.0025.16). The author is a Research Associate of this institution.

References

1. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models.
2. Yves Bestgen. 2016. Using collocational features to improve automated scoring of EFL texts.
3. Yves Bestgen. 2018. Beyond single-word measures.