# The ADAPT System Description for the STAPLE 2020 ### English-to-Portuguese Translation Task **Rejwanul Haque, Yasmin Moslem and Andy Way** ADAPT Centre School of Computing Dublin City University Dublin, Ireland firstname.lastname@adaptcentre.ie --- ## Abstract This paper describes the ADAPT Centre’s submission to STAPLE (Simultaneous Translation and Paraphrase for Language Education) 2020, a shared task of the 4th Workshop on Neural Generation and Translation (WNGT), for the English-to-Portuguese translation task. In this shared task, the participants were asked to produce high-coverage sets of plausible translations given English prompts (input source sentences). We present our English-to-Portuguese machine translation (MT) models that were built applying various strategies, e.g. data and sentence selection, monolingual MT for generating alternative translations, and combining multiple n-best translations. Our experiments show that adding the aforementioned techniques to the baseline yields an excellent performance in the English-to-Portuguese translation task. ## 1 Introduction The ADAPT Centre participated in STAPLE (Mayhew et al., 2020), a shared task of the 4th WNGT which will be held at ACL 2020, in the English-to-Portuguese language direction. The task focuses on a specific use case of MT, i.e. generating many possible translations for a given input text. Such situations are usually seen on language-learning platforms (e.g. Duolingo) where the learning process includes translation-based exercises, and evaluation is done by comparing learners’ responses with a large set of human-curated acceptable translations. The shared task organizers (Duolingo) have released real language-learner data of Duolingo as training examples. We applied a number of strategies to our MT system building process, e.g. monolingual MT, extracting parallel sentences that are similar to the Duolingo’s real language-learner data from the freely available external parallel corpora, assembling n-best translations from multiple translation systems, which essentially led us to generate high-coverage sets of possible translations of the English prompts (input source sentences). The remainder of the paper is organised as follows: Section 2 explains our approaches; Section 3 details the datasets used, explains the experimental setups and presents the results with some discussions; and Section 4 concludes our work with avenues for future work. ## 2 Methodology ### 2.1 Selecting External Datasets Since the shared task organizers released training data with a limited number of prompts (only 4,000 English prompts for English-to-Portuguese translation) and allowed participants to use external data, we made use of parallel corpora from a variety of existing sources, e.g. OPUS (Tiedemann, 2012). First, we found out which corpora are similar to Duolingo’s training dataset by measuring perplexity of the source and target texts of the external datasets on the in-domain language models (LMs) (i.e. LMs were built on the Duolingo’s data). We selected those corpora whose sentences are found to be more similar to those of Duolingo’s language learning data. ### 2.2 Selecting ‘Pseudo In-domain’ Parallel Sentences from External Data In this task, we took each of the external parallel corpora, chosen following the approach described in Section 2.1, and selected top n sentence-pairs as per low cross-entropy differences over each side of the corpus (source and target). This provided us with a 'pseudo in-domain' corpus whose sentences are similar to the sentences of the Duolingo’s data in terms of domain and style. We appended the extracted ‘pseudo in-domain’ data to the STAPLE’s (Duolingo’s) training data for building different MT systems which are described later in the paper. ### 3 Experiments and Results #### 3.1 The MT system setups We chose the classical PB-SMT and emerging NMT paradigms for building our MT systems. To build our PB-SMT systems, we used the Moses toolkit. Our PB-SMT log-linear features include: (a) 4 translational features, (b) 8 lexicalised reordering probabilities, (c) 5-gram LM probabilities, (d) 5 OSM features, and (e) word-count and distortion penalties. We used a 5-gram LM trained with modified Kneser-Ney smoothing. The weights of the parameters are optimised using the margin-infused relaxed algorithm. For decoding, the cube-pruning algorithm is applied. To build our NMT systems, we used the Marian-NMT toolkit. The learning objective of the monolingual MT models is to generate alternative sequences of a language given the sentences of the same language. We show the performance of our submitted MT systems on the development set and discuss the statistical improvements in the subsequent analysis. ### 3.2 The Shared Task Data The data released by STAPLE is compiled from Duolingo’s language learning courses. The training data for English-to-Portuguese translation contains 4,000 English prompts, leading to a total of 526,467 source–target segment-pairs. We randomly sampled 2,000 sentence-pairs from the training set for the development set. The development set released by STAPLE contains 500 English prompts which we considered as our test set in evaluating our MT systems. ### 3.3 The baseline MT systems We computed the BLEU score to evaluate the MT systems on the test set. Note that we used translations by Amazon Translate provided by STAPLE as the reference translations. Additionally, the performance of the NMT systems showed significant improvements when compared to PB-SMT systems in terms of BLEU scores, indicating better translation quality under varying data conditions. ### 3.5 The MT systems built using external datasets We built PB-SMT systems on the training data augmented by appending external data. The BLEU scores of the best system are shown in the tables detailing the performance evaluation. The systematic addition of these external datasets leads to substantial performance improvements in BLEU scores for both PB-SMT and NMT systems. ### 3.7 Generating Translations of English Prompts Our translation framework is expected to generate high-coverage sets of plausible translations given the English prompts. The main scoring metric for evaluation is the weighted macro F1. We detail the performance results of the system configurations, leading to notable improvements in coverage and precision achieved across the developed models. ## Conclusion The translation system developed for STAPLE 2020 demonstrated effective methods for generating high-coverage sets of translations through the integration of diverse data sources and machine learning techniques. Our best experimental setup earned us a competitive position in the competition, highlighting the potential of the approaches used for enhancing machine translation tasks in educational contexts.