Growing Together: Modeling Human Language Learning With n-Best Multi-Checkpoint Machine Translation

Authors

El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, Hasan Cavusoglu
Natural Language Processing Lab, Sauder School of Business, The University of British Columbia
Email: fmuhammad.mageed, moatez.nagoudig@ubc.ca, cavusoglu@sauder.ubc.ca

Abstract

We describe our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE) (Mayhew et al.,2020). We view MT models at various training stages (i.e., checkpoints) as human learners at different levels. Hence, we employ an ensemble of multi-checkpoints from the same model to generate translation sequences with various levels of fluency. From each checkpoint, for our best model, we sample n-Best sequences (n = 10) with a beam width = 100. We achieve 37:57 macro F1 with a 6 checkpoint model ensemble on the official English to Portuguese shared task test data, outperforming a baseline Amazon translation system of 21:30 macro F1 and ultimately demonstrating the utility of our intuitive method.

1 Introduction

Machine Translation (MT) systems are usually trained to output a single translation. However, many possible translations of a given input text can be acceptable. This situation is common in online language learning applications such as Duolingo, Babbel, and Busuu. In applications of this type, learning happens via translation-based activities while evaluation is performed by comparing learners’ responses to a large set of human acceptable translations. Figure 1 shows an example of a typical situation extracted from the Duolingo application.

1.1 Challenges in Machine Translation

1. Limited training data: One challenge for training a sufficiently effective model we faced is the limited size of the source training data released by organizers (4; 000 source English sentences coupled with 226; 466 Portuguese target sentences). We circumvent this limitation by training a model on a large dataset acquired from the OPUS corpus.
2. Paraphrase via MT: The shared task is a mixture of MT and paraphrase, which poses another challenge – there is no paraphrase dataset to train the system on. We generate multiple translation hypotheses (n-Best) using a wide beam search.
3. Diverse outputs: The target Portuguese sentences provided for training by organizers are produced by learners of English at various levels of fluency, making some translations inarticulate. To generate translations matching different levels of learner fluency, we employ an ensemble of checkpoints.

2 Related Work

We focus our related work overview on the task of paraphrase generation and its intersection with machine translation. Paraphrasing is expressing the same textual units in alternative forms using different words while keeping the original meaning intact. Over the last few years, MT has been the dominant approach for paraphrase generation.

Table 1: English sentences with their Portuguese translation samples from shared task training split.

3 Data Collection

3.1 Shared Task Data

As part of the STAPLE 2020 shared task, only training data were released. The target training split is a total of 526; 466 learner translations of 4; 000 input English sentences. We note that the number of translations of each English sentence varies, with an average of 132 Portuguese target sentences for each English source sentence.

Table 2: English-Portuguese datasets used in our training.

3.3 Pre-Processing

In order to develop efficient English-Portuguese MT models that can possibly work across different text domains, we utilize a large dataset of parallel English-Portuguese sentences extracted from the Open Parallel Corpus Project (OPUS).

4 Models

4.1 Architecture

Our models are based on a Convolutional Neural Network (CNN) architecture. This architecture is structured with 20 layers in the encoder and 20 layers in the decoder, a multiplicative attention in every decoder layer, and exploits BPE (Byte Pair Encoding).

4.2 Basic En→Pt Models

We trained two MT models, English-to-Portuguese (En→Pt) and Portuguese-to-English (Pt→En), on 4 V100 GPUs. The learning rate was set to 0:25, a dropout of 0:2, and maximum tokens of 4; 000 for each mini-batch.

4.3 En→Pt Extended Model

We created a new En-Pt parallel dataset using the training data of the STAPLE 2020 shared task, resulting in a new training set of 251; 442 En-Pt parallel sentences.

5 Model Deployment Methods

In order to enhance the 1-to-n En-Pt translation, we propose three methods: n-Best prediction, multi-checkpoint translation, and paraphrasing.

5.1 n-Best Prediction

We utilize our MT models to generate n-Best translation hypotheses using a beam search size of 100 and select the hypothesis with the n highest score as our output.

5.2 Paraphrasing

We use the Pt→En model to get n-Best English translations for each of the Portuguese sentences, generating new sentences that act as paraphrases.

5.3 Multi-Checkpoint Translation

We save the models at given epochs during training and use the last checkpoints to generate the n-Best translation hypotheses.

6 Evaluation

We report the performance of our methods on the official training and development datasets of STAPLE. The weights of translations correspond to user response rates and are used primarily for scoring.

7 Conclusion

In this work, we described our contribution to the 2020 Duolingo Shared Task. Our models effectively utilize n-Best prediction and multi-checkpoint translation methods. Future work will explore other methods on new language pairs.

Acknowledgements

MAM gratefully acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences Research Council of Canada (SSHRC), and Compute Canada.

References

Important authors & titles omitted for brevity