park.staple20.pdf

POSTECH Submission on Duolingo Shared Task

1,2
Junsu Park1, Hongseok Kwon1, Jong-Hyeok Lee
Department of Computer Science and Engineering1, Graduate School of Artificial Intelligence2
Pohang University of Science and Technology (POSTECH), Republic of Korea
fjspak3, hkwon, jhleeg@postech.ac.kr

Abstract

This paper describes POSTECH’s submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE) for the English-Korean language pair. In this paper, we propose a transfer learning based simultaneous translation model by extending BART. We pretrained BART with Korean Wikipedia and a Korean news dataset, and fine-tuned it with an additional web-crawled parallel corpus and the 2020 Duolingo official training dataset. In our experiments on the 2020 Duolingo test dataset, our submission achieves 0.312 in weighted macro F1 score, and ranks second among the submitted En-Ko systems.

1 Introduction

Simultaneous Translation And Paraphrase for Language Education (STAPLE) is the task of automatically producing multiple translations from a single source sentence (Mayhew et al., 2020). Because STAPLE can be regarded as a mixture of the machine translation (MT) and paraphrasing problem, MT and paraphrasing techniques play an important role in this task. Unlike in a typical MT task, systems are demanded to generate high-coverage sets on a sentence-level, as opposed to word-level. Subsequently, systems require a deeper linguistic understanding of the target language to generate accurate target sentences. Recent NLP studies have alleviated this problem.

Devlin et al. (2018) proposed bidirectional encoder representations from Transformers (BERT), trained for the reconstruction of natural language from sentences containing masked tokens, in order to obtain deeper representations for natural language. By training on an enormous amount of training data, they achieved state-of-the-art results on eleven NLP tasks. To take advantage of both pre-trained generative models and pre-trained bidirectional encoders, Lewis et al. (2019) introduced a denoising autoencoder for pre-training sequence-to-sequence models called BART. BART aims to learn linguistic knowledge in the process of first corrupting the text using various noise functions and then restoring it, and showed state-of-the-art performance in various tasks.

Given this background, we expected that using a transfer learning-based simultaneous translation model by extending BART would be effective. Our model added a randomly initialized source-side encoder in place of the embedding layer of BART pre-trained by Korean monolingual data and predicts translation weights with an additional feed-forward network using hidden vectors generated by the pre-trained decoder. The remainder of the paper is organized as follows: Section 2 describes our proposed method. Section 3 summarizes the experimental procedure and results, and Section 4 gives the conclusion.

2 Method

We adopt BART to the STAPLE problem, which takes source sentence to generate multiple target sentences. Our model consists of a pre-trained autoencoder with the source-side encoder proposed in Lewis et al. (2019) and a feed-forward network to predict translation weights (Figure 1). In the following subsections, we describe our methods in detail.

2.1 Pre-trained Autoencoder

We used BART as our pre-trained autoencoder structure. As was with BART, our autoencoder structure learns linguistic information of the target language by denoising various types of document corruptions. Among the five document corruption types proposed by BART, we applied Text Infilling and Sentence Permutation because they yielded the best results.

2.2 Source-side Encoder

Dataset Sentence Word
Monolingual 31,654,593 447,754,804
Additional parallel 2,035,566 29,964,677
Official(1 to 1) 700,410 2,915,939

2.3 Feed-forward Network for Translation Weight Training

We added a feed-forward network to predict a translation weight on each generated sentence. The sum of hidden vectors which generated on the decoder is passed as the input of the feed-forward network. The output of the feed-forward network passed through a sigmoid layer becomes the final translation weight. During the generation step, the sentences with the high weights are selected.

3 Experiments

3.1 Training Details

Fine-tuning. For fine-tuning, we used the STAPLE official training data (Duolingo, 2020) (700K sentences), setting aside 100 sentences each for the development set and test set. In addition, we adopted the web crawling parallel corpus (2M sentences) as additional training and development data for the source-side encoder. As with the pre-training corpus, we filtered out any training or development samples longer than 100 tokens.

Decoding Option Weighted Macro F1↑ Weighted Recall↑ Precision↑
Beam Size Diverse Nbest(weight)
Beam search 50 - 50 0.3192 0.3092 0.5202
75 - 75 0.3280 0.3651 0.4628
100 - 100 0.3234 0.4008 0.4214
140 - 140 0.3108 0.4394 0.3680
500 - 500 0.2218 0.5817 0.1865
Diverse beam search 100 5 100 0.1673 0.2069 0.2212
100 10 100 0.1164 0.1474 0.1601
Beam search with weight 75 - 50 0.2695 0.2546 0.4630
75 - 65 0.3064 0.3197 0.4615
75 - 70 0.3163 0.3410 0.4596

3.2 Submission Results

Systems Weighted Macro F1↑ Weighted Recall↑ Precision↑
jbrem 0.4035 0.4518 0.4795
jspak3(ours) 0.3116 0.3342 0.4701
sweagraw 0.2553 0.3168 0.3216
jindra.helcl 0.2058 0.1935 0.3894
STAPLE_fairseq_baseline 0.0486 0.0315 0.2204
STAPLE_aws_baseline 0.0412 0.0226 0.6360

4 Conclusion

Submission results. The submission results on the official test set are reported in Table 3. We selected the decoding option obtained by applying beam search with beam size 75, Nbest 75 which showed the highest weighted macro F1 score in Table 2 as our final submission. Our submission achieves an improvement of +0.263 in weighted macro F1 score compared to the baseline. As a result, our system ranks second out of the four systems submitted this year.

References