nomoto.staple20.pdf

Meeting the 2020 Duolingo Challenge on a Shoestring

Tadashi Nomoto National Institute of Japanese Literature

10-3 Midori Tachikawa 190-0014 Japan

beyond recognition. While VAE is a popular ap-

VAE, which we built in a response to the 2020 Duolingo Challenge. Both are neural models

that aim at disrupting a sentence representation the encoder generates with an eye on increasing the diversity of sentences that emerge out of the process. Importantly, we decided not to turn to external sources for extra ammunition, curious to know how far we can go while confining ourselves to the data released by Duolingo (Mayhew et al., 2020). gFCONV works by taking over a pre-trained sequence model, intercepting the output its encoder produces on its way to the decoder. c-VAE is a conditional variational auto-encoder, seeking the diversity by blurring the representation that the encoder derives. Experiments on a corpus constructed out of the public dataset from Duolingo, containing some 4 million pairs of sentences, found that gFCONV is a consistent winner over c-VAE though both suffered heavily from a low recall.

Introduction

A major driver for our participating in the challenge was the curiosity to see whether recent ap-

proaches to sentence encoding with the variational

auto-encoder (VAE) have any relevance to the generation of diverse sentences. (Bowman et al., 2016) were the first to explore the use of VAE in language generation. The work demonstrated that VAE provides a continuous code space for sentences, where any randomly picked data point in the space can be decoded to yield a coherent sentence, which is significant given that the conventional RNNs do not provide such a capability. The problem with VAE however, is that it has no mechanism to ensure that the meaning of the source sentence is passed over to the output, which often causes a sentence to be altered, or deformed

has been found on how to control or tame what it spews out. A typical solution is to fuse a VAE code with the output of a regular sentence encoder, in order to encourage the decoder to output a sentence that retains some semantic features present in the source sentence (Gupta et al., 2017). Also noteworthy is a recent work by (Guu et al., 2018), who building on an idea similar to VAE, talk about modeling the distribution of cosine similarities between word vectors for the input and target. (Li et al., 2015) is something of an odd ball in the pursuit of the diversity in sentence generation. The authors argued that we could achieve the diversity by discouraging the decoder to select candidates that are similar to the input. A clear advantage they have over others is that their scheme does not involve any learning and is straightforward to implement. The idea that one can view a latent representation as a sample drawn from some probabilistic distribution inspired people to explore its potential in a wide range of tasks and domains. (Miao et al.,

2015), while working on document modeling, suggested that we use VAE as a way to get a compact representation for a document. (Fang et al., 2019) argued for using a sample based distribution over Gaussian distribution for a latent code to better express the holistic property of the source sentence. In this work, we focus on two approaches, both based on VAE: one that attempts to achieve the diversity by generalizing the sentence representation produced by the encoder; and another which

randomly perturbs the encoder’s output during the sentence generation. We report here their respective performance on a test corpus we carved out of the official training data. For the final submission, we went along with the latter approach.

effort revolved around two questions:

2 Translation as Paraphrase

how best to incorporate likelihood scores of tar- get translations that were provided as part of the

training data, and (2) how not to rely on an external resource while building a solution. We wanted to know how far we can go using only the data made available to us at the competition, and nothing more. Our answer to the first question takes advantage of the fact that a set of translations associated with each English prompt are considered an equivalence class in the sense that if we take any pair from the set, we can substitute one for the other without significantly affecting its meaning. We may take the likelihood that a human picks a particular sentence (call it X) as a good translation for some prompt (P) as the probability of its being a paraphrase of some other sentence (say Y) from a group of possible translations of which X is part. The intuition here is that if X is more typical as a translation of P, it is more likely to serve as a paraphrase of whatever other way we may have to express P in the target language. Following this idea, we created training data by randomly sampling a pair of sentences (both in the same language) that appear as alternate translations for a given prompt in accordance with their popular rating. For each prompt, we sampled 2,000 pairs of translations (which may include pairs consisting of identical sentences), resulting in 4,601,000 training instances (which amount to 2,300 prompts plus those provided in the development and test set) (Mayhew et al., 2020). We set aside 100

1 For this year’s challenge, we worked only on the English-Japanese track. We included both test and development sets as part of training data, as a way to prevent the algorithm from stumbling upon unknown tokens in the test set. We don’t see this as much of a problem because each prompt

100 for testing. We included in each training instance an English prompt as well as its transla- tion in order to prevent paraphrases the algorithm generates from diverging from the meaning of the prompt (Table 1).

Figure 1 shows a schematic picture of how our approach works. We feed into the system a prompt and its translation which we assume to be given (via AWS, for example). Out comes its paraphrases (or translations in varied styles). The model we built is essentially one based on Fairseq’s convolution to sequence architecture of the type called ‘fconv iwslt de en’ (call it FCONV) which features 4 convolutional layers

2 for the encoder and 3 for the decoder. The embedding dimension for the input and output token was set to 256. We did not use pre-trained embeddings for either of the languages we dealt with. Neither did we make any architectural change to FCONV. We simply trained it as it was given. A departure comes in the testing phase. Following (Guu et al., 2018), we applied a Gaussian noise on the output of the encoder as it was sent to the decoder (Fig. 2).

$$ \boldsymbol{u}=E(\boldsymbol{x})+\epsilon,,,\epsilon\sim\mathcal{N}(0,k) $$

where x is an input and E(x) is an output from applying an encoder E on x. u denotes an input to a

decoder. A larger noise means a greater disruption

in the latent representation coming from the encoder, which we hoped would lead to an increase in the diversity of sentences being generated. We randomly sampled a noise from a normal distribu- tion with the mean set to 0 and the variance rang-

2 https://github.com/pytorch/fairseq

$$ E(x) $$

in development and test sets carries no more than one translation, i.e. a training pair we get from the development and test set has a same sentence for both source and target. We made use of MeCab for tokenizing sentences in Japanese.

²https://github.com/pytorch/fairseq

Table 1: Training Instances. The source part of each input consists of two sections: the first section contains a sentence in an original language, followed by a translation in a target language, demarcated by a separator '@@@@.'

| what has she done?@@@@彼女は何をやり終わったんですか？ |
| --- | --- |
| TARGET | そのりんごがとてもでかい。私は働くのが好き。かれはぎゆうにゆうを飲みます。何が彼女の強い所なの？
| 何を彼女は終わったの？ |

We also looked at a conditional variational auto- encoder (c-VAE), a close cousin of gFCONV for the sake of comparison. While both aim at building a latent representation that embraces the no- tion of uncertainty, c-VAE differs from the variance-based approach in that it seeks to find a probabilistic distribution that defines a range of representations that the encoder churns out. In terms of formulae, this comes to the following (also see Fig. 3 for a visual intuition).

$$ u=E(x)+r*z; $$

Here z = + ϵ with ϵ Unif[0;1). and are a mean and variance, defined as = g(x), and

$$ u=E(x)+r*z, $$

are

$$ z=\mu+\epsilon*v $$

some arbitrary functions over x. E(x) again denotes the output of an encoder. and are learnable parameters, which means that they need to be trained to have them work. It is worth noting that

gFCONV has no extra ‘learnable’ parameters. r is a hyper-parameter to be set manually, which determines the degree of contribution of z to a latent representation of x. We combine E(x) and a representation sampled *from a Gaussian distribution to build a *final encoder output. Our decision to

$$ \epsilon\sim\mathrm{U n i f}[0,1) $$

$$ \mu=g(x) $$

$$ \upsilon $$

often transforming it beyond recognition. Conditioning VAE on the input is a popular trick to discourage the algorithm from straying too far away from the source.

Implementation-wise, c-VAE was based on FCONV, from which we also built gFCONV. We kept all the hyper-parameter settings intact, e.g. the number of layers, the size and the number of

$$ E(x) $$

filters, etc. We did not apply any scheduled annealing weight to the KL term in the loss function. For gFCONV, we varied the variance parameter k (Eqn. 1) from 0.00 to 0.60 in increments of 0.05. For each value of k, we ran gFCONV on the test set 100 times, letting the model output 80 alter- native translations for each prompt (Setting k to 0 reduces gFCONV to a vanilla FCONV). This had resulted in a pool of 8,000 candidates for a given prompt under a particular value of k. Out of which we retained only those that had a non zero similar-

3 namely,

4 similarity to gold translations by AWS. We measured the

similarity using LASER, along with pre-trained word embeddings from FastText, which LASER requires. We were interested to know how variance affected the performance, in particular how it contributed to improving the diversity.

3 i.e. those found in the ‘test.en ja.aws baseline.pred.txt’ in the ‘staple-2020-test-blind’ directory.

https://fasttext.cc/docs/en/

⁴https://github.com/facebookresearch/LASER

⁵https://fasttext.cc/docs/en/crawl-vectors.html

| UNWEIGHTED | WEIGHTED | | | --- | --- | --- | --- | --- | --- | --- | --- | | k | P | R | Micro F1 | Macro F1 | R | Micro F1 | Macro F1 | | 0.00 | 39.16 | 3.83 | 6.97 | 11.37 | 13.62 | 20.21 | 16.35 | | 0.05 | 33.24 | 5.09 | 8.83 | 12.77 | 15.60 | 21.23 | 17.13 | | 0.10 | 28.10 | 6.18 | 10.14 | 13.46 | 17.47 | 21.54 | 17.38 | | 0.15 | 23.41 | 7.45 | 11.30 | 13.66 | 19.34 | 21.18 | 16.92 | | 0.20 | 20.08 | 8.62 | 12.06 | 13.73 | 21.10 | 20.58 | 16.48 | | 0.25 | 16.93 | 9.57 | 12.22 | 13.18 | 22.37 | 19.27 | 15.40 | | 0.30 | 14.17 | 10.24 | 11.89 | 12.27 | 23.36 | 17.64 | 14.13 | | 0.35 | 12.02 | 10.69 | 11.31 | 11.22 | 24.16 | 16.05 | 12.74 | | 0.40 | 10.92 | 11.49 | 11.20 | 10.70 | 24.70 | 15.14 | 12.03 | | 0.45 | 9.15 | 11.23 | 10.08 | 9.67 | 24.38 | 13.30 | 10.73 | | 0.50 | 8.21 | 10.81 | 9.33 | 8.72 | 23.24 | 12.14 | 9.72 | | 0.55 | 7.24 | 9.99 | 8.40 | 7.90 | 22.84 | 11.00 | 8.78 | | 0.60 | 6.92 | 9.30 | 7.94 | 7.54 | 22.25 | 10.56 | 8.38 |

3 Results and Discussion

the following discussion, we concentrate on un- k = 0 : 00 indicates wild ups and downs in perfortifully well on some but failing miserably on oth- shown were produced using the official scorer. In

weighted scores as our interest here is in knowing how much we improved the raw recall under the current setup. Note that weighted scores do not shed light on the true diversity of sentences we have garnered. Looking at Table 2, we see gFCONV gaining on a vanilla FCONV, whose performance is represented by the numbers at k = 0: 00. At k = 0: 25, we see the raw recall jumping from 3.83 to 9.57,

Micro F1 from 6.97 to 12.22, and Macro F1 from 11.37 to 13.18. Compare the difference between Micro and Macro F1 at k = 0*:* 00 and that we have at k = 0*:* 25. The difference for the latter is much smaller. This suggests that under gFCONV, the performance is more stable across test items compared to the vanilla FCONV. A large divergence at

ers. In contrast to Micro F1, Macro F1 is blind to how many candidate translations there are for each prompt, so may not give us an accurate picture of how the model is doing on each prompt. As with gFCONV, we ran c-VAE on the test set 100 times, obtaining 100 distinct pools of candi- date translations for each prompt. We report in Table 3, figures that represent performance on all

6 We generated 8,000 hypotheses for each prompt under a particular value of r, 80 at each round.

Phase	Rank	Precision	W. Recall	W.F1
DEV	6/6	0.369	0.183	0.181
TEST	6/6	0.349	0.212	0.194

official evaluation was prepared using gFCONV at k = 0*:* 10, under the pseudonym ‘darkside,’ with 7 the official results shown in Table 4.

4 Conclusions

We discussed two approaches as a way to tackle the Duolingo Challenge. One is gFCONV, which

takes over a pre-trained sequence model,

cepts and perturbs the output its encoder produces on its way to the decoder. Another is c-VAE, a conditional variational auto-encoder, which seeks the diversity by blurring the representation that the encoder derives. Either approach, it was found, outperformed the vanilla FCONV. We also noted a large discrepancy between Micro and Macro F1, suggesting that the models’ performance is not even and fluctuates wildly from item to item. Moreover, there were some test prompts for which the models were not able to find any translations. We recognize that this is an area we need to scrutinize to further improve the performance. In the long run, it would be interesting to see if we can bring to the task recent developments in VAE such as (Bouchacourt et al., 2018).

References

Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 2018. Multi-level variational autoen-

coder: Learning disentangled representations from grouped observations. In

Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Con- ference on Computational Natural Language Learn- ing, pages 1021, Berlin, Germany. Association for Computational Linguistics. Le Fang, Chunyuan Li, Jianfeng Gao, Wen Jun Dong, and Changyou Chen. 2019. Implicit deep

generative framework for paraphrase generation. CoRR, abs/1709.05074.

Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437450. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. CoRR, abs/1510.03055. S. Mayhew, K. Bicknell, C. Brust, B. McDowell, W. Monroe, and B. Settles. 2020. Simultaneous translation and paraphrase for language education. In Proceedings of the ACL Workshop on Neural Generation and Translation (WNGT). ACL. Yishu Miao, Lei Yu, and Phil Blunsom. 2015. Neural variational inference for text processing. CoRR, abs/1511.06038.

We did not submit the version at 0.25 which turned out to be the best, due to its late discovery, which came well past the deadline.