# Training and Inference Methods for High-Coverage Neural Machine Translation

**Michael Yang**  
**Rahul Mayuranath**  
Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA, U.S.A.  
fmyang2,yixinl2,rmayurang@cs.cmu.edu

## Abstract
In this paper, we introduce a system built for the Duolingo Simultaneous Translation And Paraphrase for Language Education (STAPLE) shared task at the 4th Workshop on Neural Generation and Translation (WNGT 2020). We participated in the English-to-Japanese track with a Transformer model pretrained on the JParaCrawl corpus and finetuned in two steps on the JESC corpus and then the (smaller) Duolingo training corpus. First, during training, we find it is essential to deliberately expose the model to higher-quality translations more often during training for optimal translation performance. For inference, encouraging a small amount of diversity with Diverse Beam Search to improve translation coverage yielded marginal improvement over regular Beam Search. Finally, using an auxiliary filtering model to filter out unlikely candidates from Beam Search improves performance further. We achieve a weighted F1 score of 27.56% on our own test set, outperforming the STAPLE AWS translations baseline score of 4.31%.

## Introduction
Currently, state of the art machine translation systems generally produce a single output translation. However, human evaluators of translation tasks will often accept multiple translations as correct. We introduce a neural machine translation (NMT) system that generates high-coverage translation sets for a single given prompt in the source language.

1. Introduction consist of English prompts and a weighted set of target language translations for each prompt. The task requires systems to produce translation sets for given English prompts that are evaluated on weighted F1 score, defined in Appendix A. We have made our code publicly available. We experimented with models trained and finetuned on the provided Duolingo English-Japanese prompt-translation data (Mayhew et al., 2020), the JParaCrawl web-crawled corpus (Morishita et al., 2019), as well as the Japanese-English Subtitle Corpus (JESC) (Pryzant et al., 2018). The sizes of each dataset are summarized in Table 1.

Our system uses a Transformer-based (Vaswani et al., 2017) NMT model and we began with weights pretrained on the large JParaCrawl corpus (Morishita et al., 2019). Section 4 describes in detail how the model was pretrained. Our system’s NMT model was then obtained by fine-tuning first on the Japanese-English Subtitle Corpus (JESC) (Pryzant et al., 2018) before further fine-tuning on the Duolingo training set (Mayhew et al., 2020). We outline these datasets in more detail in Section 2.

Outputting the entire beam of candidates from 150-width Beam Search, scored on per token log likelihood, this two-step fine-tuned system produced the translations that we submitted to the shared task leaderboard. It achieved 25.69% weighted F1 score on the shared task blind development set and 26.0% on the blind test set. After the leaderboard closed, we conducted further experiments and discovered several notable optimizations. The most effective optimization was using the ground truth weights that indicate variations in translation quality during training. We find that it is essential to deliberately expose the model to higher-quality translations more often during training. Otherwise, overexposure to low-quality translations harms the model’s translation performance. Secondly, Diverse Beam Search with a very small penalty outperformed Beam Search. However, too much diversity begins to introduce minor semantic shifts that deviate from correct translations. We also explored introducing an auxiliary filtering model for post-processing candidates. Our proposed filtering model is able to refine the candidates generated by the NMT model, which improved the system’s performance with respect to the weighted F1 score.

We share our results in Section 7. Our best result was a weighted F1 score of 27.56% on our own test set of 200 prompts randomly selected from the training data.

### 2.1 Duolingo High-coverage Translations
Duolingo provided training, development and test sets (Mayhew et al., 2020). However, the development and test datasets were ‘blind’ and did not contain ground truth translations, so we did not use these for training or development. The training set consists of 2,500 English sentences from which we learned corresponding target translations.

| Dataset | English Sentences | Japanese Sentences |
| --- | --- | --- |
| JParaCrawl | 8,763,995 | 8,763,995 |
| JESC | 2,801,388 | 2,801,388 |
| Duolingo | 2,500 | 855,940 |

**Table 1:** Number of sentence-pairs in the datasets (Duolingo pairs have a one-to-many correspondence)

### 2.2 JParaCrawl
As our base model, we use a model pre-trained on the JParaCrawl corpus (Morishita et al., 2019). This corpus contains over 8.7 million sentence pairs which were crawled from the web and then automatically aligned, similar to European corpora in the ParaCrawl project. Though noisy due to an imperfect alignment method, this is currently the largest publicly-available English-Japanese bitext corpus.

### 2.3 Japanese-English Subtitle Corpus
The Japanese-English Subtitle Corpus (JESC) (Pryzant et al., 2018), is a large parallel training corpus that contains 2.8 million pairs of TV and movie subtitles. With an average length of 8, the corpus mostly consists of short sentences, which is similar to the data present in the Duolingo training corpus. Even though JESC contains some noise, it captures sufficient information that is useful for downstream NMT tasks.

### 3 Related work
There are various neural approaches to tackle machine translation. These include utilizing recurrent neural networks (Cho et al., 2014b), convolutional neural networks (Kalchbrenner et al., 2016), attention-based models (Luong et al., 2014; Bahdanau et al., 2015) and transformer networks (Vaswani et al., 2017). Sequence to sequence models deal with the task of mapping an input sequence to an output sequence. These were first introduced by Sutskever et al. (2014) and typically use an RNN-based encoder-decoder architecture, where the encoder outputs a fixed length representation of the input which is fed into the decoder to get a target translation.

### The Transformer Architecture
For our experiments, we used the Transformer architecture proposed by Vaswani et al. (2017). It is a self-attention based model that produces superior results for machine translation tasks compared to CNN and LSTM based models. By stacking multiple layers of multi-head self-attention blocks, they demonstrate that the attention mechanism by itself is very powerful for sequence encoding and decoding. Recently, Transformer-based models that are pre-trained on large-scale datasets have produced superior performance on various Natural Language Processing (NLP) tasks.

### 4 Pretrained Base Model
As our base model, we used a model pretrained by Morishita et al. (2019) on the JParaCrawl data using the fairseq framework.

### 5 Filtering Model
Apart from the NMT model, we additionally introduce a neural filtering model to post-process the NMT model’s candidates. Instead of designing a model that will assign a real-value score to each of the candidates, we simplify the task by formulating it as a binary classification problem. Namely, the filtering model is trained to classify a given candidate sentence as a valid sample (in the gold-standard list) or an invalid sample. The intuition is that the gold-standard candidate list contains a small number of high-quality sentences (with larger weights) and a large number of lower-quality sentences. Thus it is more important to distinguish the hits from misses than high-quality hits from low-quality hits.

### 6 Experiments
We experiment with several different fine-tuning scenarios, each time evaluating the models using the Weighted F1 metric on our 200-prompt Duolingo test set. First as a baseline, we directly evaluate the JParaCrawl pretrained model without fine-tuning. Then we evaluate the performance of models fine-tuned on either JESC or on all English-Japanese pairs in our 2,100-prompt Duolingo training set. Finally, we experiment with first fine-tuning on the JESC data and then on the Duolingo training set.

| Fine-tuning | Precision | Recall | Weighted Recall | Weighted F1 |
| --- | --- | --- | --- | --- |
| None(JParaCrawl only) | 18.57% | 4.62% | 16.97% | 14.23% |
| JESC | 13.37% | 3.49% | 13.12% | 10.69% |
| Duolingo | 34.59% | 10.48% | 26.90% | 24.96% |
| JESC+Duolingo | 35.98% | 10.89% | 27.85% | 25.92% |

### 7 Results
We conducted our experiments sequentially and generally used the best results so far as a baseline for subsequent experiments. Our best performing model was the one trained using multi-step fine-tuning, as shown in Table 2. The performance of this model was superior to the other fine-tuning settings on every metric, suggesting this result was not simply a matter of imbalance between precision and recall.

---

As for loss smoothing, contrary to our hypothesis, increasing the loss smoothing rate was detrimental. and, in fact, decreasing the rate from 0.1 to 0.05 even improved the weighted F1 score slightly from 27.21% to 27.43%. This suggests that the effect of loss smoothing on the high-coverage translation task is not necessarily different to the usual machine translation task.

### 8 Conclusions and Future Work
Our machine translation system produces high-coverage sets of target language translations from single source language prompts. We used multi-step fine-tuning to train a robust NMT model. This involved first training or finetuning a model on a large bitext dataset, then fine-tuning on the bitext dataset with high coverage sets of target language translations. One idea for future work is to directly optimize the weighted F1 score during training using reinforcement learning.