The JHU Submission to the 2020 Duolingo Shared Task on Simultaneous Translation and Paraphrase for Language Education

Huda Khayrallah, Jacob Bremerman, Arya D. McCarthy, Kenton Murray, Winston Wu, and Matt Post
Center for Language and Speech Processing, Johns Hopkins University
Human Language Technology Center of Excellence, Johns Hopkins University
University of Maryland, College Park

Abstract

The Duolingo 2020 STAPLE Shared Task (Mayhew et al., 2020) focuses on generating a comprehensive set of translations for a given sentence, translating from English into Hungarian, Japanese, Korean, Portuguese, and Vietnamese. The formulation of this task differs from the conventional machine translation setup: instead of the n-gram match (BLEU) against a single reference, sentence-level exact match is computed between a list of proposed candidates and a weighted list of references. The set of references is drawn from Duolingo’s language-teaching app. Any auxiliary data is allowed for building systems, including existing very-large parallel corpora for translation. Our approach begins with strong MT systems.

1 Introduction

The objective of the task is to recover the list of references, and performance is measured by a weighted F-score. Entries outperform baseline weighted F1 scores by a factor of 2 to 10 and are ranked first in the official evaluation for every language pair.

In addition to our system description, we perform additional analysis. We find that stronger BLEU performance of the beam-search generated translation is not indicative of improvements on the task metric—weighted macro F1 of a set of hypotheses—and suggest this should encourage further research on how to train NMT models when n-best lists are needed. We perform detailed analysis on our output, which led to additional development on English–Portuguese.

2 Task Description

Table 1: Statistics over the Duolingo-provided data.

These translations come from courses intended to teach English to speakers of other languages; the references are initially generated by trained translators, and augmented by verified user translations. Each translation is associated with a relative frequency denoting how often it is selected by Duolingo users. All of the provided task data is lower-cased.

For each language pair, we created an internal split of the Duolingo-provided training data: 100 training prompts for use in validating the MT system (JHU-VALID), another 100 intended for model selection (JHU-DEV), and a 300-prompt test set for candidate selection (JHU-TEST). The remaining data (JHU-TRAIN) was used for training the MT models.

Weighted F1

Weighted Macro F1 is computed as follows:

[ F_{1} = \sum_{s \in S} \frac{\text{Weighted} F_{1}(s)}{|S|},]

where S is all prompts in the test corpus. The weighted F1 is computed with a weighted recall, where TPs are the true positives for a prompt s, and FNs are the false negatives for a prompt s.

Evaluation

Evaluation is conducted on lowercased text with the punctuation removed.

3 Machine Translation Systems

We began by building high-quality state-of-the-art machine translation systems.

Data and Preprocessing

Additional data for our systems was obtained from Opus. We removed duplicate bitext pairs, then reserved 3k random pairs from each dataset to create validation, development, and test sets of 1k sentences each. The validation dataset is used as held-out data to determine when to stop training the MT system.

Table 2: Number of English word tokens for all datasets used to train the baseline MT models.

Training Models

We used fairseq to train standard Transformer models with several configurations. We trained two sets of models: MODEL1 and MODEL2, with fine-tuning strategies varying based on data selections.

Table 3: The weighted macro F1 on JHU-TEST for MODEL1 and fine-tuned variants.

Results

We present results of our different methods on our internal development set and present our official evaluation performance in subsequent sections.

Conclusion

Our approach was general, borrowing from best practices in machine translation. We built large, general-domain MT systems that were then fine-tuned on in-domain data. We then followed an "overgenerate and filter" approach that made effective use of the scores from the systems to find a per-prompt truncation of large n-best lists produced from these systems. These techniques performed very well, ranking first in all five language pairs.

Acknowledgments

We thank Najoung Kim for feedback on Korean, An Nguyen for remarks on Vietnamese and Vinicius C. Costa for input on Portuguese.