# 2020 Duolingo Shared Task

## STAPLE: Simultaneous Translation And Paraphrase for Language Education

This challenge is in conjunction with the [WNGT](https://sites.google.com/view/wngt20/home) workshop at [ACL 2020](https://acl2020.org/).

# Introduction

Machine translation systems typically produce a **single output**, but in certain cases, it is desirable to have **many possible translations** of a given input text. This situation is common with [Duolingo](/content/site-root.html) (the world's largest language-learning platform), where some learning happens via translation-based exercises, and grading is done by comparing learners' responses against a large set of human-curated acceptable translations. We believe the processes of grading and/or manual curation could be vastly improved with richer, multi-output **translation + paraphrase systems**.

In this shared task, participants start with English prompts and generate high-coverage sets of plausible translations in five other languages. For evaluation, we provide sentences with handcrafted, field-tested sets of possible translations, weighted and ranked according to actual learner response frequency. We will also provide high-quality automatic translations of each input sentence that may (optionally) be used as a reference/anchor point, and also serves as a strong baseline. In this way, we expect the task to be of interest to diverse researchers in machine translation, MT evaluation, multilingual paraphrase, and language education technology fields.

Novel and interesting research opportunities in this task:

- A large set of sentences with **comprehensive** translations (though not exhaustive, per se)
- Translations weighted by **real language-learner data**
- Datasets in **5 language pairs**.

The outcomes of this shared task will be:

- New **translation datasets** provided to the community
- New **benchmarks** for MT and paraphrasing

## Official Results

**(Updated May 26, 2020)** The table below shows the overall results on the TEST set for each language, in terms of weighted F1. We have omitted results that did not outperform the baselines. More detailed results can be found in [this spreadsheet](https://docs.google.com/spreadsheets/d/18_3-9CKKn1sxRSmNIC7lMRzSHc4PPAeuxUfU38ULkNI/edit?usp=sharing).

These results differ slightly from the [CodaLab leaderboard](https://competitions.codalab.org/competitions/23643#results). CodaLab chooses to display a results from a team's entire submission (according to some comparison function). We have chosen to select **each team's highest performing score** for each language track, which are not necessarily all from the same submission.

| user | hu | ja | ko | pt | vi |
| --- | --- | --- | --- | --- | --- |
| [jbrem](http://sharedtask.duolingo.com/papers/khayrallah.staple20.pdf) | 0.555 | 0.318 | 0.404 | 0.552 | 0.558 |
| [nickeilf](http://sharedtask.duolingo.com/papers/li.staple20.pdf) | -- | -- | -- | 0.551 | -- |
| [rakchada](http://sharedtask.duolingo.com/papers/chada.staple20.pdf) | 0.552 | -- | -- | 0.544 | -- |
| [jspak3](http://sharedtask.duolingo.com/papers/park.staple20.pdf) | -- | -- | 0.312 | -- | -- |
| [sweagraw](http://sharedtask.duolingo.com/papers/agarwal.staple20.pdf) | 0.469 | 0.294 | 0.255 | 0.525 | 0.539 |
| [masahiro](http://sharedtask.duolingo.com/papers/kaneko.staple20.pdf) | -- | 0.283 | -- | -- | -- |
| [mzy](http://sharedtask.duolingo.com/papers/yang.staple20.pdf) | -- | 0.260 | -- | -- | -- |
| [dcu](http://sharedtask.duolingo.com/papers/haque.staple20.pdf) | -- | -- | -- | 0.460 | -- |
| [jindra.helcl](http://sharedtask.duolingo.com/papers/libovicky.staple20.pdf) | 0.435 | 0.213 | 0.206 | 0.412 | 0.377 |
| [darkside](http://sharedtask.duolingo.com/papers/nomoto.staple20.pdf) | -- | 0.194 | -- | -- | -- |
| [nagoudi](http://sharedtask.duolingo.com/papers/nagoudi.staple20.pdf) | -- | -- | -- | 0.376 | -- |
| [baseline_aws](http://sharedtask.duolingo.com/papers/mayhew.staple20.pdf) | 0.281 | 0.043 | 0.041 | 0.213 | 0.198 |
| [baseline_fairseq](http://sharedtask.duolingo.com/papers/mayhew.staple20.pdf) | 0.124 | 0.033 | 0.049* | 0.136 | 0.254* |

## Important Dates

|     |     |
| --- | --- |
| **July 10, 2020** | Workshop at ACL — VIRTUAL ( [Link](https://sites.google.com/view/wngt20/home),<br> [Registration](https://acl2020.org/registration/))! |
| **May 18, 2020** | Camera-ready system papers due |
| **May 4, 2020** | System paper reviews returned |
| April 13 **April 22, 2020** | Draft system papers due ( [SoftConf START](https://www.softconf.com/acl2020/wngt/)) |
| April 8 **April 16, 2020** | Final results announcement |
| April 6 **April 13, 2020** | Final predictions deadline |
| **March 30, 2020** | Data release (phase 3): blind TEST set ( [Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/38OJR6),<br> [CodaLab](https://competitions.codalab.org/competitions/23643)) |
| **March 2, 2020** | Data release (phase 2): blind DEV set ( [Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/38OJR6),<br> [CodaLab](https://competitions.codalab.org/competitions/23643)) |
| **January 13, 2020** | Data release (phase 1): TRAIN set + starter code ( [Dataverse](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/38OJR6),<br> [GitHub](https://github.com/duolingo/duolingo-sharedtask-2020/)) |

## Task Definition & Data

### Background

Duolingo is a free, award-winning, online language learning platform. Since launching in 2012, more than 300 million students from all over the world have enrolled in one of Duolingo's 90+ game-like language courses, via the website or mobile apps. For comparison, that is more than the total number of students in the entire U.S. school system.

A portion of learning on Duolingo happens through translation-based exercises. In this task, we focus on the challenges where users are given a prompt in the language they are learning (English), and type a response in their native language. Some examples of this are shown in the following images, which are taken from English lessons for Portuguese speakers.

### Prediction Task

Participants are given an English sentence, and are required to produce a high-coverage set of translations in the target language. In order to level the playing field, we also provide a high-quality automatic reference translation (via Amazon), which may be considered as a strong baseline for the machine translation task.

The prompt sentences come from Duolingo courses, and are often relatively simple (and a little quirky). For example, below is a sentence taken from the course that teaches English to Portuguese speakers:

|     |     |
| --- | --- |
| **INPUT: Prompt** | is my explanation clear? |
| **INPUT: Reference Translation (from Amazon)** | a minha explicação está clara? |
| **OUTPUTS: Accepted Translations** | minha explicação está clara?<br>minha explicação é clara?<br>a<br> minha explicação está clara?<br>a minha explicação é clara?<br>minha<br> explanação está clara?<br>está clara minha explicação?<br>minha<br> explanação é clara?<br>a minha explanação está clara?<br>é<br> clara minha explicação?<br>a minha explanação é clara?<br>está<br> clara a minha explicação?<br>é clara a minha explicação?<br>está<br> clara minha explanação?<br>é clara minha explanação?<br>está<br> clara a minha explanação?<br>é clara minha explanação |

Examining these data, it’s clear that not all accepted translations are equally likely, and therefore, they should be scored accordingly. As stewards of the world's largest and most comprehensive corpus of language learning data, we are able to use lesson response data to estimate which translations are more likely. This is used in the metric, described below.

### Data

The data for this task comes from five Duolingo courses. All use English prompts, with multiple translations, although weighted by frequency from speakers of each of the following languages:

- `en_pt` — **Portuguese**
- `en_hu` — **Hungarian**
- `en_ja` — **Japanese**
- `en_ko` — **Korean**
- `en_vi` — **Vietnamese**

The TRAIN data will include comprehensive accepted translations along with weights for participating teams to use in tuning their systems.

Statistics for the released training data are below. All dev and test sets have the same number of prompts (500). The training sets are sampled from each course.

| Language | TRAIN (prompts / total accepted) | DEV (prompts / total accepted) | TEST (prompts / total accepted) |
| --- | --- | --- | --- |
| en_hu | 4000 / 251442 | 500 / 27647 | 500 / 33578 |
| en_pt | 4000 / 526466 | 500 / 60294 | 500 / 67865 |
| en_ja | 2500 / 855941 | 500 / 172817 | 500 / 165095 |
| en_ko | 2500 / 700410 | 500 / 140353 | 500 / 150477 |
| en_vi | 3500 / 194720 | 500 / 29637 | 500 / 28242 |

## Submission & Evaluation

### Starter Code

You can find starter code here: [https://github.com/duolingo/duolingo-sharedtask-2020/](https://github.com/duolingo/duolingo-sharedtask-2020/). This contains code to train standard seq2seq models, as well as the official scoring function, and some data readers.

### Metrics

The main scoring metric will be weighted macro F1, with respect to the accepted translations. In short, systems are scored based on how well they can return **all human-curated acceptable translations**, weighted by the **likelihood that an English learner would respond** with each translation.

In weighted macro F1, we calculate weighted F1 for each prompt s, and take the average over all prompts in the corpus. We chose to calculate precision in an unweighted fashion, and weight only recall. Specifically, for weighted true positives (WTP) and weighted false negatives (WFN), we have:

WTPs=∑t∈TPsweight(t)WFNs=∑t∈FNsweight(t)WeightedRecall(s)=WTPsWTPs+WFNs

The weighted F1's are then averaged over all prompts in the corpus.

Weighted Macro F1=∑s∈SWeighted F1(s)|S|

### Evaluation: CodaLab

All system submissions and evaluation will be done via [CodaLab](https://competitions.codalab.org/). There will be a DEV phase where you can submit predictions online after the phase 2 data release, and a TEST phase for final evaluation after the phase 3 data release. Check for more details as the submission deadline approaches.

### Prediction Format

The submission file format is similar to the Amazon Translate prediction file.

Submission should have blocks of text separated by one empty line, where the first line of the block is an ID and prompt, and all following lines are unique predicted paraphrases, order doesn't matter. During evaluation, the punctuation will be stripped, and all text lowercased.

Here is an example prediction file, with prompts corresponding to the example above:

```
prompt_65c64c31d672de7ed6e084757731dc60|is my explanation clear?
minha explicação está clara?
minha explicação é clara?
a minha explicação está clara?
a minha explicação é clara?
an obviously incorrect prediction!

prompt_5665133396132783ba2fd30154c44ab0|this is my fault.
isto é minha culpa.
isso é minha culpa.
esta é minha culpa.
isto é culpa minha.
é minha culpa.
essa é minha culpa.
isso é culpa minha.
isto é minha falta.
esta é minha falta.
isto é minha falha.
essa é minha falha.
my bad.
```

### System Papers & Citation Details

All teams are expected to submit a system paper describing their approach and results, to be published in the workshop proceedings and available through the [ACL Anthology](https://aclanthology.info/) website. Please do so even if you are unable to travel to the ACL conference in July 2020.

Note that we are interested not only in top-performing systems (i.e., metrics), but also _meaningful findings_ (i.e., insights for language and/or learning). Teams are encouraged to focus on both in their write-ups!

Papers should follow the the [ACL 2020 submission guidelines](https://acl2020.org/calls/papers/). Teams are invited to submit a full paper (4-8 pages of content, with unlimited pages for references). We recommend using the official style templates:

- [LaTeX + MS Word](https://acl2020.org/downloads/acl2020-templates.zip)
- [Overleaf](https://www.overleaf.com/latex/templates/acl-2020-proceedings-template/zsrkcwjptpcd)

All submissions must in **PDF format** and should **not be anonymized**. Supplementary files (hyperparameter settings, external features or ablation results too extensive to fit in the main paper, etc.) are also welcome, so long as they follow the ACL 2020 Guidelines. Final camera ready versions of accepted papers will be given up to one additional page of content (9 pages plus references) to address reviewer comments. Papers must include the following citation:

> Stephen Mayhew, Klinton Bicknell, Chris Brust, Bill McDowell, Will Monroe, and Burr Settles. 2020. Simultaneous Translation And Paraphrase for Language Education. In _Proceedings of the ACL Workshop on Neural Generation and_ _Translation (WNGT)_, ACL.

```
@inproceedings{staple20,
  Author = {Stephen Mayhew and Klinton Bicknell and Chris Brust and Bill McDowell and Will Monroe and Burr Settles},
  Booktitle = {Proceedings of the ACL Workshop on Neural Generation and Translation (WNGT)},
  Publisher = {ACL},
  Title = {Simultaneous Translation And Paraphrase for Language Education},
  Year = {2020}}
```

### Tips, Resources, & Related Work

The following resources may prove useful. We may update this section as the challenge progresses....

## Translation

- [fairseq](https://github.com/pytorch/fairseq) is a Pytorch-based framework for sequence modeling, such as machine translation or text generation.

- [KyTea may be useful for segmentation in Japanese.](http://www.phontron.com/kytea/)
- Multilingual contextual models, many of which are available through [HuggingFace transformers.](https://github.com/huggingface/transformers)

- [Multilingual BERT](https://github.com/google-research/bert/blob/master/multilingual.md) has proven to be remarkably useful for cross-lingual applications.

- [XLM](https://github.com/facebookresearch/XLM) (Lample & Conneau, 2019), and [XLM-R](https://arxiv.org/abs/1911.02116) (Conneau et al., 2019), by virtue of parallel text in training, may outperform Multilingual BERT.

- It may be important to maintain a [diverse beam when decoding](https://www.aclweb.org/anthology/P19-1365/) (Ippolito et al., 2019).

- Given the nature of the data, phrase-based systems such as [Giza++](https://github.com/moses-smt/giza-pp), [Moses](http://statmt.org/moses/), and others may in fact be competitive with more modern, neural methods (see [statmt.org](http://statmt.org/) for many resources)

## MT Evaluation

- In the [HyTer](https://www.aclweb.org/anthology/N12-1017/) metric (Dreyer & Marcu, 2012), a translation prediction is scored against a comprehensive list of manually-gathered translations. Our evaluation is similar in the sense that we have high-coverage translation options at test time, but the goals are slightly different. Where HyTer is concerned with accurate measurement of machine translation, this task pursues high-coverage output. We also provide real-world weights with each translation option.

- Where HyTer employed humans in writing all possible translations of a sentence, [Automated HyTer](https://www.cis.upenn.edu/~ccb/publications/hyter-paraphrase-lattices.pdf) (Apidianaki et al., 2018) uses the [Paraphrase Database (PPDB)](http://paraphrase.org/).

## Paraphrasing

- The [Multilingual Paraphrase Database (PPDB)](http://paraphrase.org/#/download) may prove useful (Ganitkevitch & Callison-Burch, 2014)

- [Towards Universal Paraphrastic Sentence Embeddings](https://arxiv.org/abs/1511.08198) (Wieting et al., 2015)

- [Simple and Effective Paraphrastic Similarity from Parallel\ Translations](https://www.aclweb.org/anthology/P19-1453.pdf) (Wieting et al, 2019)

- [Learning to Paraphrase: An Unsupervised Approach Using\ Multiple-Sequence Alignment](https://arxiv.org/pdf/cs/0304006.pdf) (Barzilay & Lee, 2003)

## MT Corpora

Participants are encouraged to use all available resources for supplemental training, including:

- [Opensubtitles](https://opus.nlpl.eu/OpenSubtitles/corpus/version/OpenSubtitles)
- [Tatoeba](https://tatoeba.org/eng/)
- [Europarl](https://www.statmt.org/europarl/)
- [News Commentary](http://www.casmacat.eu/corpus/news-commentary.html)
- The [OPUS Collection](http://opus.nlpl.eu/)