# Exploring Model Consensus to Generate Translation Paraphrases

**Zhenhao Li**, **Marina Fomicheva**, **Lucia Specia**  
1 Department of Computing, Imperial College London  
2 Department of Computer Science, University of Sheffield  
  
fzhenhao.li18, l.speciag@imperial.ac.uk  
fm.fomichevag@sheffield.ac.uk

## Abstract
This paper describes our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE). This task focuses on improving the ability of neural MT systems to generate diverse translations. Our submission explores various methods, including N-best translation, Monte Carlo dropout, Diverse Beam Search, Mixture of Experts, Ensembling, and Lexical Substitution. Our main submission is based on the integration of multiple translations from multiple methods using Consensus Voting. Experiments show that the proposed approach achieves a considerable degree of diversity without introducing noisy translations. Our final submission achieves 0.5510 weighted F1 score on the blind test set for the English-Portuguese track.

## 1 Introduction
In the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE) (Mayhew et al., 2020), participating MT systems are evaluated using multiple reference translations to measure their ability to generate diverse, yet high quality translations. For that, a new dataset with multiple human translations for each source sentence is provided. These human translations were produced by language learners as part of a translation exercise on the Duolingo platform where they were asked to translate sentences from the language they were learning (e.g. English) to their native language. Each translation in the dataset is assigned a weight based on the learner response frequency. The STAPLE dataset includes five language pairs: English to Portuguese, Hungarian, Japanese, Korean, and Vietnamese. In the shared task, we only participated in English-Portuguese (En-Pt) track.

| Original | is my explanation clear? |
| --- | --- |
| Translation | minha explicação está clara? | 0.2673 | minha explicação é clara? | 0.1616 | a minha explicação está clara? | 0.1111 | a minha explicação é clara? | 0.0878 | minha explicação está clara? | 0.0572 | está clara minha explicação? | 0.0443 | minha explicação é clara? | 0.0392 |

**Table 1:** An example of weighted translations in the STAPLE dataset for English-Portuguese.

In this paper, we experiment with various methods to improve the diversity of translations, while preserving their quality. We show that simply by generating N-best translations with larger beam size, we can achieve a considerable degree of diversity. Our final submission is based on the integration of multiple translations from various methods, namely N-best translation, Monte Carlo dropout, Mixture of Experts, Ensembling, and Lexical Substitution, through a consensus voting mechanism. It achieves 0.5510 weighted F1 score on the official blind test set.

## 2 Methods
In what follows we describe the methods used in our experiments, including N-best translation, Monte Carlo dropout, Diverse Beam Search, Mixture of Experts, Ensembling and Lexical Substitution. We combine all of these methods except the Diverse Beam Search in our official submissions through a consensus voting mechanism. Details about the submissions can be found in Section 5.

### 2.1 N-best
The simplest method to generate multiple translations for a given sentence is to use N-best translations with a large beam size during decoding. Larger beam size might lead to more translation options with similar meanings. We experimented with multiple sizes for N and used the same value for N-best and beam size.

### 2.2 MC Dropout
Gal and Ghahramani (2016) proposed the Monte Carlo (MC) dropout method to estimate predictive NMT model uncertainty. The method consists in running several forward passes through the model (i.e., at inference time), each applying dropout before every weight layer and collecting posterior probabilities generated by the model with parameters perturbed by dropout. The mean and variance of the resulting distribution can then be used to represent model uncertainty. Instead of using this method for scoring translations, we use it as a way to generate alternative MT hypotheses for a given source sentence. Specifically, we run inference with dropout M times and collect the resulting translations. In our experiments, the dropout rate is set to 0.1 and M = 10.

### 2.3 Diverse Beam Search
Vijayakumar et al. (2016) proposed the Diverse Beam Search algorithm to improve the diversity of beam hypotheses. The algorithm proceeds by dividing the beam budget into groups and enforcing diversity between groups of beams. In our experiments we use the implementation of this algorithm in fairseq (Ott et al., 2019) with default parameters.

### 2.4 Mixture of Experts
Training an ensemble of various MT models initialized with different random seeds is a common strategy used to boost the output quality (Garmash and Monz, 2016). Unlike the typical ensembling method that combines prediction distributions from different models by averaging, we use each system in the ensemble to generate a separate set of translation hypotheses, and take the set of distinct translations as the final output.

### 2.5 Lexical substitution
In the STAPLE dataset, we observed that many of the paraphrases in translations are simple variants with word substitutions in the target language. Therefore, we built a dictionary containing all lexical substitutions from the STAPLE training data. The substitutions are sorted according to two criteria: 1) number of occurrences 2) substitution probability.

**Table 2:** Top-5 lexical substitutions in frequency-sorted and probability-sorted dictionaries.

| Frequency |  | Probability |  |
| --- | --- | --- | --- |
| substitution | count | substitution | prob |
| neste-nesse | 5091 | baixar-＞descarregar | 1.0 |
| irá-vai | 4920 | descarregar-＞baixar | 1.0 |
| vou-irei | 4645 | situa-se-＞fica | 1.0 |
| local-lugar | 2989 | achasse-＞encontrasse | 1.0 |
| bem-bastante | 2694 | localizasse-＞achasse | 1.0 |

### 2.6 Consensus voting
To integrate translations from different models, we employed a consensus voting mechanism by counting the number of systems that predicted each translation. A threshold Tcon is set, meaning that a translation must be predicted by at least Tcon + 1 systems, otherwise it is removed. Considering the lexical translation might generate rare but correct translation, we assign the lexical-substituted translations a weight Wsub so that they can be seen as generated by Wsub systems. The consensus method guarantees a high precision by removing translations that are likely to be incorrect.

## 3 Experiments
To build the NMT model, we used parallel corpora for En-Pt from OPUS (Tiedemann, 2012) as out-of-domain data, including ParaCrawl, EUbookshop, Europarl, Wikipedia, QED, and Tatoeba. The combination of these corpora contains 22.42 million parallel sentence pairs. The STAPLE dataset, which contains 4000 source sentences with 526,466 translations, is used as in-domain data for fine-tuning.

## 3.1 Data Processing
All sentences are tokenized with Moses (Koehn et al., 2007), and then processed via Byte-Pair-Encoding (BPE) (Sennrich et al., 2016). A shared vocabulary of 40,000 subwords is constructed for both English and Portuguese. The training data was cleaned by removing sentence pairs with more than 250 subwords or with length ratio over 1.5, using the clean-corpus-n.perl script in Moses.

## 3.2 Model and Hyperparameters
We used the Transformer model (Vaswani et al., 2017) as our baseline model. The model is trained using fairseq toolkit (Ott et al., 2019) with the default hyperparameter settings using transformer_wmt_en_de architecture. The model was trained on 8 GPUs with a batch size of 4096 tokens on each GPU. We used mixed-precision training to accelerate the training. The model was pre-trained on OPUS data for 30 epochs and then fine-tuned on STAPLE data. We set 5 as the number of experts for training the MoE system.

## 3.3 Generation of Translations
1. Generate translations from N systems, resulting in N translation sets s1; s2; s3; :::; sN  
2. Apply consensus voting to the N system translations with threshold Tcon, resulting in one translation set sconsensus  
3. Apply lexical substitution to sconsensus, resulting in a separate translation set slexical  
4. Apply consensus voting to the N system translations and the lexical substitution translations with threshold Tcon and weight Wsub, resulting in the final translation set slexical&consensus.

## 4 Results
The translations are evaluated at sentence level as a classification problem where true positives (TP) occur when the system produces one of the translations in the given set of references, false positives (FP) when a translation out of this set is produced, and false negatives (FN) when translations in this set are missed by the system. The official evaluation metric is a weighted macro F1-score averaging over all source sentences.

| ID | System | Precision | Recall | F1 | Hyperparameters |
| --- | --- | --- | --- | --- | --- |
| 1 | nbest | 0.714 | 0.484 | 0.521 | N=12, tune-10 |
| 2 | ensemble | 0.617 | 0.549 | 0.523 | N=12, tune-10, 3 pretrained * 4 fine-tune seeds |
| 3 | nbest | 0.645 | 0.522 | 0.518 | N=18, tune-10 |
| 4 | nbest | 0.635 | 0.522 | 0.511 | N=20, tune-20 |
| 5 | MoE | 0.368 | 0.527 | 0.385 | N=12, tune-10, experts=5 |
| 6 | MC Dropout | 0.660 | 0.496 | 0.514 | N=3, tune-10, M=50 |
| 7 | MC Dropout | 0.672 | 0.485 | 0.511 | N=5, tune-10, M=10 |
| 8 | Consensus | 0.653 | 0.534 | 0.530 | 12 * nbest systems (tune-10), Tcon=1 |
| 9 | Consensus | 0.641 | 0.541 | 0.529 | 12 * nbest systems (tune-10), 2 MC Dropout systems (row 6 and 7), 5 experts, Tcon=2 |
| 10 | Consensus | 0.677 | 0.527 | 0.536 | same as Row 9, Tcon=3 |
| 11 | Lexical | 0.443 | 0.538 | 0.428 | same as Row 10, add lexical substitutions (frequency > 4000) |
| 12 | Lexical | 0.612 | 0.534 | 0.509 | same as Row 11, frequency > 5000 |
| 13 | Consensus | 0.633 | 0.565 | 0.538 | 24 * nbest systems (tune-10, tune-20), 2 MC Dropout systems, 5 experts, Tcon=4 |
| 14 | Consensus | 0.652 | 0.558 | 0.542 | same as Row 13, Tcon=5 |
| 15 | Consensus | 0.655 | 0.557 | 0.543 | 24 * nbest systems (tune-10, tune-20), 2 MC Dropout systems, 3 experts, Tcon=5 |
| 16 | Lexical | 0.607 | 0.578 | 0.533 | same as Row 15, add lexical substitution (probability > 0.85) |
| 17 | Lexical+Consensus | 0.651 | 0.561 | 0.544 | same as Row 15, add lexical substitution to consensus voting (Wsub=3) |
| 18 | Lexical+Consensus | 0.667 | 0.553 | 0.546 | same as Row 17, Tcon=6 |
| 19 | Lexical+Consensus | 0.682 | 0.546 | 0.548 | same as Row 17, Tcon=7 |
| 20 | Lexical+Consensus | 0.697 | 0.540 | 0.550 | same as Row 17, Tcon=8 |
| 21 | Consensus | 0.710 | 0.530 | 0.550 | same as Row 15, Tcon=9 |
| 22 | Consensus | 0.721 | 0.526 | 0.551 | same as Row 15, Tcon=10 |
| 23 | Lexical only | 0.243 | 0.114 | 0.127 |  |
| 24 | Lexical+Consensus | 0.720 | 0.526 | 0.550 | same as Row 22, add lexical substitutions to consensus voting (frequency > 1000, Wsub=3) |
| 25 | Consensus | 0.564 | 0.582 | 0.506 | 36 * nbest systems (tune-10, tune-20, tune-40), 2 MC Dropout systems, 3 experts, Tcon=10 |
| 26 | Consensus | 0.618 | 0.565 | 0.526 | same as Row 25, Tcon=11 |
| 27 | Lexical+Consensus | 0.722 | 0.527 | 0.552 | same as Row 22, add lexical substitutions to consensus voting (probability > 0.99, Wsub=3) |
| 28 | Lexical+Consensus | 0.720 | 0.529 | 0.552 | same as Row 27, Wsub=7 |

## 6 Conclusions
This paper describes our submissions to the STAPLE shared task for English-Portuguese translation. Our best submission achieves the second position in the English-Portuguese track, with only 0.0006 weighted F1 score behind the winning submission. We showed that simply generating N-best translations already achieves a considerable degree of diversity and quality. We experimented with various methods to improve the diversity in the MT output, including N-best translation, MC Dropout, Diverse Beam Search, Mixture of Experts, Ensembling, Consensus Voting, and Lexical Substitution. We showed the benefits and drawbacks of these methods in generating diverse, high quality translations. Our systems combining these methods further improve over the N-best translation and achieve 0.5510 weighted F1 score on STAPLE blind test set, which is only 0.0006 behind the winning submission.