Generating Diverse Translations via Weighted Fine-tuning and Hypotheses Filtering for the Duolingo STAPLE Task

Sweta Agrawal
Department of Computer Science
University of Maryland
sweagraw@cs.umd.edu

Marine Carpuat
Department of Computer Science
University of Maryland
marine@cs.umd.edu

Abstract

This paper describes the University of Maryland’s submission to the Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). Unlike the standard machine translation task, STAPLE requires generating a set of outputs for a given input sequence, aiming to cover the space of translations produced by language learners. We adapt neural machine translation models to this requirement by (a) generating n-best translation hypotheses from a model fine-tuned on learner translations, oversampled to reflect the distribution of learner responses, and (b) filtering hypotheses using a feature-rich binary classifier that directly optimizes a close approximation of the official evaluation metric. Combination of systems that use these two strategies achieves F1 scores of 53.9% and 52.5% on Vietnamese and Portuguese, respectively ranking 2 and 4 on the leaderboard.

1 Introduction

While machine translation (MT) typically produces a single output for each input, scoring and generation for second language learning applications might benefit from systems whose outputs better capture the diversity of translations produced by language learners. The Duolingo Simultaneous Translation And Paraphrase for Language Education (STAPLE) shared task provides a framework for developing and testing such systems, grounded in real translations produced by English learners into five native languages (Portuguese, Vietnamese, Hungarian, Japanese, Korean). In this task, given an English sentence prompt, systems are asked to produce a set of translations for that prompt, and are scored based on how well their outputs cover human-curated acceptable translations, weighted by the likelihood that an English learner would respond with each translation (Table 1).

Table 1: STAPLE data: given a prompt in English, translation alternatives are weighted according to Learner Response Frequency (LRF).

2 Background

Unlike in the STAPLE task, recent attempts at generating multiple translations for a single source have targeted output variability along specific stylistic dimensions or produce diverse outputs without a specific use case. The techniques used can be divided into three categories: (a) constrain the decoding process to generate diverse candidates; (b) optimize via a diversity promoting loss function; (c) expose the model to different translation candidates with side-constraints or without. Since it is unclear what dimensions of variations are captured in the STAPLE translation, we focus instead on improving n-best lists generated by a standard neural MT model.

3 Approach

3.1 Frequency-Aware Hypotheses Generation

Given the STAPLE data for a language pair, where the i-th training example includes a source sentence in English, a reference set of K translations and corresponding LRF weights, we create MT training samples by copying the translation pair (e_i, f_j), w O times. Given model parameters, this yields a weighted cross-entropy loss: [ \mathcal{L}{lrf}(\theta)=\sum{i=1}^{M}\sum_{j=1}^{K}(w_{i}^{j} \times O)\log P(f_{i}^{j}|e_{i};\theta) ]

3.2 Hypothesis Filtering as Binary Classification

Even when informed by STAPLE data and LRF scores, n-best lists might include translations that are not in the reference set, due to translation errors or selecting paraphrases that do not match language learners’ preferences. We design a binary classifier that further filters the n-best lists by predicting for each hypothesis whether or not it should be included in the final set. This lets us define features based on the complete prompt and hypothesis sequence pairs, while the MT model generates the hypothesis incrementally.

4 Experiment Settings

4.2 MT configurations

Model Architecture: We use the Transformer model implemented in the Sockeye toolkit as a baseline MT system.

4.3 Filtering configurations

Classifier: The 2-layer feed-forward NN has 5 hidden units and 2 output units.

5 Evaluation

We evaluate the lowercased detokenized output of the systems on our internal test dataset using:

• Weighted Macro F1: the official scoring metric which quantifies how the set of system outputs covers the human-curated acceptable translations, weighted by the LRF of each translation.

6 Experiment Results

6.1 Impact of Frequency-Aware Fine-Tuning

Fine-Tuning: The Frequency-Aware n-best hypotheses consistently yield the best Weighted Recall and Weighted Macro-F1 scores for all languages.

7 Submitted Systems

8 Conclusion

We proposed two strategies to obtain multiple outputs that mimic translations by produced by language learners from a standard neural MT model. Our experiments showed that (1) fine-tuning MT models using all reference translations and their weight yields more diverse n-best hypotheses that better reflect learner preferences, and (2) filtering these n-best lists using a feature-rich classifier trained to maximize an approximation of the STAPLE evaluation metric yields further improvements. Combinations of systems that use these two strategies approach the top scoring submission in the official evaluation.

References

• Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation.
• Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.
• Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation.
• Kyunghyun Cho. 2016. Noisy parallel approximate decoding for conditional recurrent language model.