Simultaneous Translation and Paraphrase for Language Education

Stephen Mayhew, Klinton Bicknell, Chris Brust, Bill McDowell, Will Monroe, and Burr Settles
Duolingo
Pittsburgh, PA, USA
{stephen, klinton, chrisb, mcdowell, monroe, burr}@duolingo.com

Abstract

We present the task of Simultaneous Translation and Paraphrasing for Language Education (STAPLE). Given a prompt in one language, the goal is to generate a diverse set of correct translations that language learners are likely to produce. This is motivated by the need to create and maintain large, high-quality sets of acceptable translations for exercises in a language-learning application, and synthesizes work spanning machine translation, MT evaluation, automatic paraphrasing, and language education technology.

We developed a novel corpus with unique properties for five languages (Hungarian, Japanese, Korean, Portuguese, and Vietnamese), and report on the results of a shared task challenge which attracted 20 teams to solve the task. In our meta-analysis, we focus on three aspects of the resulting systems: external training corpus selection, model architecture and training decisions, and decoding and filtering strategies. We find that strong systems start with a large amount of generic training data, and then finetune with in-domain data, sampled according to our provided learner response frequencies.

1 Introduction

Machine translation systems are typically trained to produce a single output, but in certain cases, it is desirable to have many possible translations of a given input text. For example, Duolingo—the world’s largest language-learning platform—uses translation-based exercises for some of its lessons. For any given translation prompt there may be hundreds or thousands of valid responses, so we use a set of human-curated translations in order to grade learner responses. The manual process of maintaining these sets is laborious, and we believe it can be improved with the aid of rich multi-output translation and paraphrase systems.

Table 1: An example from the Portuguese dataset. In this task, teams are given an English prompt and a reference translation, and are required to produce as many variants in the accepted translations as possible. The evaluation favors translations with higher weight, which is a measure of learner response frequency.

To this end, we introduce a new task called Simultaneous Translation and Paraphrasing for Language Education (STAPLE). From the perspective of the research community, we believe this poses an interesting exercise that is similar to machine translation (MT), but also provides data with new and unique properties that we expect to be of interest to researchers in MT evaluation, multilingual paraphrasing, and even language education technology.

field-tested accepted translations, each weighted and ranked according to their empirical frequency among Duolingo learners. We also provide a high-quality automatic reference translation of each prompt that may (optionally) be used as a reference or anchor point, in the event that researchers want to explore paraphrase-only approaches (this also serves as a strong baseline). See Table 1 for an example from the Portuguese dataset.

2.1 Corpus Collection

One such format is a translation exercise, in which the learner is shown a prompt in one language, and asked to translate it into the other. Since English is by far the most popular language to learn on Duolingo, we created a task corpus by sampling prompts from English courses, in which users are shown an English sentence, and then asked to translate it into a language they already know. For instance, the examples come from the course for Portuguese speakers learning English.

Naturally, some prompts have more accepted translations (valid learner responses) than others, depending on such factors as polysemy, synonymy, or prompt length. We filtered out prompts for which the number of accepted translations was in the top or bottom deciles of a course, to avoid outliers. Although each accepted translation is technically correct, usually a small number of them are considered most fluent or idiomatic. To estimate this distribution empirically, we gathered learner response data from October–November 2019. For each translation, we counted the number of times that learners produced that translation (with some allowances for punctuation and capitalization). This provided a count for each translation in the set of accepted translations.

Table 2: Dataset sizes by number of prompt sentences, and total number of accepted translations.

We provide data for translating English prompts into five languages: Hungarian, Japanese, Korean, Portuguese (Brazilian), and Vietnamese. These span five different language families, three different writing systems, and represent a wide variety of popular Duolingo courses. For the shared task, participants were allowed to submit results to any or all of these language tracks. Furthermore, there were no restrictions on the use of external data; teams were encouraged to use any available monolingual or parallel corpora.

Table 3: F1 results for all systems, on all languages. Rank is assigned according to statistical significance.

Official weighted F1 results are shown in Table 3. Ranks are determined using an approximate permutation test, and adjacent-scoring systems are considered significantly different at p < .05. Overall, teams outperformed our provided baselines by a wide margin, and submissions tended to score higher on precision than weighted recall.

6 Conclusion and Future Work

We learned that a pipeline of strong machine translation followed by fine-tuning on learner-weighted STAPLE data produces strong results. While the data for this task are geared toward language learners (and are therefore simpler than more commonly-studied domains such as newswire), it is our hope that the STAPLE task provides a blueprint for ongoing interdisciplinary work in this vein. All task data, including dev and test labels, will remain available at: https://doi.org/10.7910/DVN/38OJR6

Acknowledgements

The authors would like to thank Colin Cherry for seeding the idea that ultimately became the STAPLE task. Thanks also to the organizers of the Workshop on Neural Generation and Translation (WNGT) for providing a forum for this work, as well as all the participating teams. Special thanks to Nathan Dalal and Andrew Runge for help reviewing and summarizing the system papers.