Expand and Filter: CUNI and LMU Systems for the WNGT 2020

Duolingo Shared Task

Authors

1. Jindrich Libovicky, Center for Information and Language Processing, LMU Munich, Germany
2. Zdenek Kasner, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic
3. Jindrich Helcl, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic
4. Ondrej Dusek, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

Abstract

We present our submission to the Simultaneous Translation And Paraphrase for Language Education (STAPLE) challenge. We used a standard Transformer model for translation, with a crosslingual classifier predicting correct translations on the output n-best list. To increase the diversity of the outputs, we used additional data to train the translation model, and we trained a paraphrasing model based on the Levenshtein Transformer architecture to generate further synonymous translations. The paraphrasing results were again filtered using our classifier. While the use of additional data and our classifier filter were able to improve results, the paraphrasing model produced too many invalid outputs to further improve the output quality. Our model without the paraphrasing component finished in the middle of the field for the shared task, improving over the best baseline by a margin of 10–22% weighted F1 absolute.

1 Introduction

The STAPLE challenge (Mayhew et al.,2020) poses the problem of MT slightly differently. Here, the goal is to generate as many correct translations as possible. Knowing many correct translations can be useful e.g. for automatic scoring in tools for language education, such as Duolingo. On one hand, the learners should be guided to use the more common formulations, on the other hand, they should not be penalized for providing a correct but unusual answer.

System Pipeline

We present a pipeline of two systems in our submission to the STAPLE challenge. As the first step, we use a standard NMT model trained with additional, carefully filtered data. The NMT output n-best lists are filtered using a classifier. Second, we use a Levenshtein Transformer model (Gu et al., 2019) to generate paraphrases of the outputs of the first model. Again, the outputs of the Levenshtein Transformer are filtered using another classifier. The paper is structured as follows. Our training datasets are described in detail in Section 2. We describe the two models in Sections 3 and 4 respectively. We conduct experiments with all five target languages in the challenge, i.e. Hungarian, Japanese, Korean, Vietnamese and Portuguese. The source language is English in all setups. The experiment settings are shown in Section 5. The results (Section 6) show that the Levenshtein Transformer paraphrase generator cannot easily improve on the filtered NMT output n-best list.

Data

2.1 STAPLE Dataset

We held out 200 source sentences with all their translations as our internal validation dataset. We use this dataset for validating the translation models and for estimating the filtering thresholds.

2.2 Additional data for MT

For training the translation model, we obtained out-of-domain parallel corpora from the OPUS collection (Tiedemann, 2012) for all target languages, ParaCrawl (Espla et al., `2019) for Portuguese and Hungarian, and JParaCrawl (Morishita et al., 2019) for Japanese. We applied FastText language identifier (Joulin et al., 2016b,a) to clean the corpora. Overall data usage and statistics are shown in Table 1 and Table 2.

Table 1: STAPLE training data statistics (target language set).

3 Translation and Filtering Model

3.1 Translation Model

Our pipeline starts with the Transformer model (Vaswani et al., 2017) trained on the provided dataset enriched with additional data (see Section 2). This provides initial translations of the source sentence on the output n-best list, which are further filtered.

3.2 Filtering Classifier

We train a crosslingual classifier which predicts whether a translation in the MT output n-best list is correct (given the source sentence). Using the trained translation model, we first generate large n-best lists for all English sentences in the original training data. Next, we label each generated sentence whether it is a positive or a negative sample based on the reference data. We create a balanced mix of samples during this process.

4 Paraphrasing Model

As an additional step to increase the number of valid translations produced, we train a target-based sequence generation model based on the Levenshtein Transformer. The model allows for sequence refinement and generation of paraphrases.

5 Experiments

5.1 Evaluation Metric

We use the official evaluation metric for the STAPLE challenge, which is the weighted macro F1 score, computed by exact match with respect to the set of all valid translations for a given source sentence.

6 Results

Experiments demonstrated that while our model with the filter was able to gain substantial increases over the baseline, it did not reach the top places in the challenge. The paraphrasing component’s output proved too noisy to bring about any substantial benefits.

7 Conclusions

We presented our submission to the 2020 STAPLE translation and paraphrasing shared task. Our model is based on the Transformer architecture, used additional carefully selected training data, a XLM-R-based classifier to filter MT output beams, and an optional paraphrasing component based on the Levenshtein Transformer approach. Despite the challenges faced, our approach achieved noticeable improvements across targeted languages.

Acknowledgements

Work conducted at CUNI was supported by the Charles University grant No. 140320, the SVV project No. 260575, the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 825303 (Bergamot), Czech Science Foundation grant No. 19-26934X (NEUREM3), and the Charles University project PRIMUS/19/SCI/10.

References

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate.
2. Lo¨ıc Barrault et al. 2019. Findings of the 2019 conference on machine translation (WMT19).
3. Ondˇrej Bojar et al. 2013. Scratching the surface of possible translations.
4. Alexis Conneau et al. 2020. Unsupervised cross-lingual representation learning at scale.
5. Miquel Espla et al. 2019. ParaCrawl: Web-scale parallel corpora for the languages of the EU.
6. Jiatao Gu et al. 2019. Levenshtein transformer.
7. Rico Sennrich et al. 2016. Improving neural machine translation models with monolingual data.
8. Ashish Vaswani et al. 2017. Attention is all you need.