Simultaneous paraphrasing and translation by fine-tuning Transformer models

Rakesh Chada
Amazon.com, Inc.
rakchada@amazon.com

Abstract

This paper describes the third place submission to the shared task Mayhew et al. (2020) on simultaneous translation and paraphrasing for language education at the 4th workshop on Neural Generation and Translation (WNGT) for ACL 2020. The shared task involves generating multiple translations for a given source text in English and a target language. The five target languages in the task are Hungarian (hu), Portuguese (pt), Japanese (ja), Korean (ko), and Vietnamese (vi). We competed in the Hungarian and Portuguese tracks. A goal of the shared task, hosted by Duolingo, is to enable the development of automated grading processes and curation systems for language learners' responses. A high-coverage and precise multi-output translation and paraphrasing system would vastly help such automated efforts. For the task, participants were provided with hand-crafted and field-tested sets of several possible translations for each English sentence. Each of these translations were also ranked and weighted according to actual learner response frequency, and these weights were provided as additional features. Along with these, translations from AWS were provided as a baseline and additional data. The challenges associated with the shared task are two-fold: i) Translating from English to target languages and ii) Producing multiple valid translations (paraphrases) while balancing precision with coverage. We conduct several experiments to address these two challenges and develop a simple system that leverages pre-trained transformer models and a wide beam search strategy. Furthermore, we leverage the provided translation scores and experiment with multiple training distribution strategies to develop a simple oversampling strategy that produces improvements over the vanilla method of using one translation one time.

1 Introduction

2 Related work

Paraphrasing and machine translation are well-studied research areas in general, but there’s not much research specifically in the context of multi-output translation systems, especially for low-resource languages. Tan et al. (2019) train a Transformer-based Neural Machine Translation model for Hungarian-English and Portuguese-English translation. However, their goal was to assess the benefits of multilingual modeling by clustering languages and is different from that of a multi-output translation system. For English-Portuguese, Aires et al. (2016) build a phrase-based machine translation system to translate biomedical texts. For multilingual paraphrasing, Ganitkevitch and Callison-Burch (2014) release a database consisting of paraphrases for several languages, including Hungarian and Portuguese, at lexical, phrasal, and syntactic levels. Guo et al. (2019) build a zero-shot multilingual paraphrase generation model to show mixed results. However, their end goal was to generate paraphrases in the same language (English) as opposed to our shared task which requires generating paraphrases in a different language.

Table 1: Dataset statistics. MSL=Maximum Source Length. MTL=Maximum Target Length. 99p SL=99th percentile Source Length. 99p TL=99th percentile Target Length.

Ippolito et al. (2019) study diverse decoding methods on conditional language models and show promising results on movie dialogue corpus and image captioning tasks.

3 Task

We describe dataset statistics and evaluation metrics in this section.

3.1 Data

There are two phases of the competition - Dev and Test. Table 1 shows data statistics for all phases. There were 4000 train prompts provided in English for both Hungarian and Portuguese languages. However, each of these prompts were accompanied by multiple translations leading to 251,442 English-Hungarian (en-hu) pairs and 526,466 English-Portuguese (en-pt) pairs. There were 500 prompts in both dev and test phases. After tokenization, for en-hu, most of the source sentences were shorter than 11 tokens and target sentences were shorter than 14 tokens. For en-pt, most of the source sentences were shorter than 25 tokens and target sentences were shorter than 15 tokens.

3.2 Evaluation Metrics

The main scoring metric for the competition is the weighted macro F1 score. This is a measure of how well the system returns all human-curated translations weighted by the likelihood that an English learner would respond with each translation. For each prompt p, weighted macro F1 is calculated as the harmonic mean of precision and weighted recall (note that the precision is unweighted). To calculate weighted recall for each example, we first calculate Weighted True Positives (WTP) and Weighted False Negatives (WFN) as:

WTP = ext{weight}(t)

WFN_{p} = extstyle ext{ ext{sum}}_{t ext{ in } FN_{p}} ext{weight}(t)

Then, weighted recall (WR) is calculated as:

WR_{p} = rac{WTP_{p}}{WTP_{p}+WFN_{p}}

The weighted Macro F1 (WF) over all prompts P is then calculated by averaging over all prompts in the corpus:

WF = extstyle ext{ ext{sum}}_{p ext{ in } P} rac{WF_{p}}{|P|}

4 System Design

We describe the final submitted system design in detail. We have experimented with several other variants and describe these in a later section.

4.1 Data Sampling

For the final system, we chose to use weighted sampling of the data where the weights correspond to the provided learner response frequency. Specifically, we multiply the frequency of the translation (a number between 0 and 1) with a heuristic value of 50 and duplicate the source-translation pair that many times. In effect, this would create repeated samples of certain pairs whose frequency is greater than 0.02 while eliminating pairs whose frequency is less than 0.02. With this sampling, we end up with 40,500 en-hu pairs and 42,000 en-pt pairs. We separate 15% of the provided prompts as a validation set. The performance on this validation set is used to pick the best model.

4.2 Preprocessing

The final submitted model architecture, shown in Figure 1, uses the standard Transformer sequence-to-sequence model. This has 6 encoder and 6 decoder layers and an 8-headed attention mechanism in both encoder and decoder. We initialize the model with the pre-trained representations obtained from the OPUS-MT data. This model is then fine-tuned on the task data. We tie the encoder, decoder, and output embedding weights and use a shared vocab size of 60,522. For position-wise feed-forward layers, the Swish activation function is used. The whole model is fine-tuned, through an early stopping mechanism, on the dataset constructed as detailed in section 4.1.

4.3 Hyperparameters

We use batch size set to 500, dropout set to 0.1, label smoothing set to 0.1, and Adam optimizer with a learning rate of 3e-4, where β1 = 0.9, β2 = 0.98 and epsilon = 1e-9. We decay the learning rate by an inverse square root mechanism for 16000 steps. The gradient clip norm is set to 5. And patience for early stopping is set to 5.

5 Results & Discussion

Table 2 shows results of the final submission for en-hu and en-pt tracks, along with a comparison to the baseline. As per the main evaluation metric (Weighted Macro F1 score), our model outperformed the strong AWS baseline by a significant margin on both en-hu and en-pt tracks. For en-hu, the improvement is about 30 absolute points on the dev dataset and 27 points on the test dataset. For en-pt, the improvement is about 34 absolute points on the dev dataset and 33 absolute points on the test dataset. This model ranked 1st on the dev leaderboard and 2nd on the test leaderboard for en-hu track. It ranked 2nd on the dev leaderboard and 3rd on the test leaderboard for en-pt track.

Table 2: Final submission results. Bold indicates best performance. P=Precision. WR=Weighted Recall. WF=Weighted Macro F1.

8 Summary

We describe the system for our submission to the shared task on simultaneous translation and paraphrasing for language education at the 4th workshop on Neural Generation and Translation (WNGT) for ACL 2020. The final submitted system leverages pre-trained translation models, with Transformer architecture, and an oversampling strategy to achieve competitive performance.