Publications

We are reinventing high stakes testing with innovative tools and techniques, supported by the latest research in language assessment, psychometrics, and machine learning.

Duolingo English Test: Technical Manual

Ben Naismith, Ramsey Cardwell, Geoffrey T. LaFlair, Steven Nydick, Masha Kostromitina (2026)

Duolingo Research Report](https://go.duolingo.com/dettechnicalmanual)


Duolingo English Test: Demographic and Score Properties of Test Takers (July 2024-June 2025)

Allison Michalowski, Ramsey Cardwell, Steven Nydick, and Ben Naismith (2025)

Duolingo Research Report](https://go.duolingo.com/demographic-score)


The Evolution of the Duolingo English Test

Masha Kostromitina (2025, updated 2026)

Duolingo Research Report DRR-26-03](http://go.duolingo.com/det-evolution)


Analytics for Quality Assurance for Item Pools (AQuAP): Monitoring and Maintaining Item Bank Health in AI-Driven Assessment Systems

Alina A. von Davier, Xiaowan Zhang, Yigal Attali, Yena Park, Jacqueline Church, Andrew Runge, Geoff T. LaFlair, Alexander Tsigler (2026)

ArXiv Pre-Print](https://doi.org/10.48550/arXiv.2606.18536)


A Scalable Parametric Item Calibration Engine (SPICE) for Explanatory IRT with Sparse Data

Steven Nydick, Manqian Liao, J.R. Lockwood (2026)

ArXiv Pre-Print](https://arxiv.org/abs/2605.21782)


A Systematic Review of Test Component Ordering with Implications for Language Assessment

Ben Naismith and Ramsey Cardwell (2026)

Language Testing](https://journals.sagepub.com/doi/10.1177/02655322261432256)


Responsible AI Standards

Jill Burstein (2026)

Duolingo Research Report](https://go.duolingo.com/ResponsibleAI)


Advancing integrity in language assessment: Response to Bruce et al.

Maria Kostromitina and Ramsey Cardwell (2026)

Advancing integrity in language assessment: Response to Bruce et al., ELT Journal, 2026;, ccaf065, https://doi.org/10.1093/elt/ccaf065](https://academic.oup.com/eltj/advance-article/doi/10.1093/elt/ccaf065/8417073)


A Theoretical Assessment Ecosystem for a Digital-First Assessment

J. Burstein, G. T. LaFlair, and A. A. von Davier (2026)

Duolingo Research Report DRR-26-02](https://go.duolingo.com/ecosystem)


Alignment of the Duolingo English Test (DET) to the Common European Framework of Reference for Languages (CEFR): An External Validation of Individual Skill Subscores for Speaking, Writing, Reading, and Listening

Kelley Stethen and Chad Buckendahl (2025)

ACS Ventures, LLC Technical Report](https://go.duolingo.com/DET-SWRL-CEFR-Technical-Report)


Where Assessment Validation and Responsible AI Meet

Jill Burstein & Geoff LaFlair (2025)

In C. Coombe, T. Clark, & H. Mohebbi (Eds.), Language Teaching Research Quarterly, 50, 120–137. https://doi.org/10.32038/ltrq.2025.50.09](https://api.eurokd.com/Uploads/Article/1883/ltrq.2025.50.09.pdf)


A Concordance Study between the DET and IELTS Academic SWRL Subscores

Steven Nydick, J.R. Lockwood (2025)

Duolingo Research Report](https://go.duolingo.com/concordance-det-ielts-2024)


Responsible Al for test equity and quality: The Duolingo English Test as a case study

Burstein, J., LaFlair, G. T., Yancey, K., von Davier, A. A., & Dotan, R. (2025)

In E. M. Tucker, E. Armour-Thomas, & E. W. Gordon (Eds.), Handbook for Assessment in the Service of Learning, Volume I: Foundations for Assessment in the Service of Learning. University of Massachusetts Amherst Library Press.](https://www.assessment-for-learning.org/volume-i)


An Overview of Duolingo English Test Administration and Scoring

Steven Nydick, J.R. Lockwood (2024, updated 2025)

Duolingo Research Report](https://go.duolingo.com/Admin+Scoring)


Exploring AI-Enabled Test Practice, Affect, and Test Outcomes in Language Assessment

Jill Burstein, Ramsey Cardwell, Ping-Lin Chuang, Allison Michalowski, and Steven Nydick (2025)

Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con 2025), Pittsburgh, PA.) - Full Papers Volume.](https://aclanthology.org/2025.aimecon-main.7.pdf)


When Machines Mislead: Human Review of Erroneous AI Cheating Signals

Will Belzak, Chenhao Niu, and Angel Ortmann Lee (2025)

Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con 2025), Pittsburgh, PA.)- Works in Progress Volume.](https://aclanthology.org/2025.aimecon-wip.11/)


Keystroke Analysis in Digital Test Security: AI Approaches for Copy-Typing Detection and Cheating Ring Identification

Chenhao Niu, Yong-Siang Shih, Manqian Liao, Ruidong Liu, and Angel Ortmann Lee (2025)

Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con 2025), Pittsburgh, PA.)- Works in Progress Volume.](https://aclanthology.org/2025.aimecon-wip.13/)


Span labeling with large language models: Shell vs. meat. \
Phoebe Mulcaire and Nitin Madnani (2025)

Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025) (pp. 850–859). Association for Computational Linguistics.](https://aclanthology.org/2025.bea-1.62.pdf)


Beyond GPA and language proficiency: A systematic literature review of international students’ academic success factors.

Masha Kostromitina, Ben Naismith, Jill Burstein, & Luke Plonsky (2025)

Review of Education](https://bera-journals.onlinelibrary.wiley.com/doi/10.1002/rev3.70089)


A multi-stage interactive writing task for the assessment of English language writing proficiency

Andrew Runge, Sarah Goodwin, Yigal Attali, Mya Poe, Phoebe Mulcaire, Kai-Ling Lo, and Geoffrey T. LaFlair (2025)

Language Testing](https://journals.sagepub.com/doi/10.1177/02655322251349908)


Interactive Listening – The Duolingo English Test

Geoffrey T. LaFlair, Andrew Runge, Yigal Attali, Yena Park, Jacqueline Church, Sarah Goodwin (2023, updated 2025)

Duolingo Research Report](https://go.duolingo.com/interactive-listening-whitepaper)


Assessing Listening on the Duolingo English Test

Sarah Goodwin, Ben Naismith (2023, updated 2025)

Duolingo Research Report](https://go.duolingo.com/listening-whitepaper)


Assessing Speaking on the Duolingo English Test

Yena Park, Ramsey Cardwell, Sarah Goodwin, Ben Naismith, Geoffrey T. LaFlair, Kai-Ling Lo, Kevin P. Yancey (2023, updated 2025)

Duolingo Research Report](https://go.duolingo.com/speaking-whitepaper)


Assessing Vocabulary on the Duolingo English Test

Yena Park, Ramsey Cardwell, Ben Naismith (2024, updated 2025)

Duolingo Research Report](http://go.duolingo.com/vocabulary)


Duolingo English Test: Writing Construct

Sarah Goodwin, Yigal Attali, Geoffrey T. LaFlair, Yena Park, Andrew Runge, Alina A. von Davier, Kevin P. Yancey, Ben Naismith, Ping-Lin Chuang (2022, updated 2025)

Duolingo Research Report](https://go.duolingo.com/scored-writing)


Developing an Automatic Pronunciation Scorer: Aligning Speech Evaluation Models and Applied Linguistics Constructs

Danwei Cai, Ben Naismith, Masha Kostromitina, Zhongwei Teng, Kevin P. Yancey, Geoffrey T. LaFlair (2025)

Language Learning](https://onlinelibrary.wiley.com/doi/10.1111/lang.70000)


Guidelines for Fair Test Content: The Duolingo English Test Example

Jacqueline Church, Yena Park, and Jill Burstein (2025)

Duolingo Research Report DRR-25-03](https://go.duolingo.com/DETFairnessGuidelines)


Duolingo English Test: Security and Score Integrity

William Belzak, Basim Baig, Ramsey Cardwell, Rose Hastings, André Kenji Horie, Geoff LaFlair, Manqian Liao, Chenhao Niu, and Yong-Siang Shih (2025)

Duolingo Research Report DRR-25-01](https://duolingo-papers.s3.us-east-1.amazonaws.com/reports/DET_Security_Report.pdf)


The Item Factory: Intelligent Automation in Support of Test Development at Scale

Alina A. von Davier, Andrew Runge, Yena Park, Yigal Attali, Jacqueline Church, & Geoff T. LaFlair (2024)

In "Machine Learning, Natural Language Processing, and Psychometrics", eds. Hong Jiao & Robert W. Lissitz](https://doi.org/10.1108/979-8-88730-606-320251002)


Where Assessment Validation and Responsible AI Meet

J. Burstein and G. T. LaFlair (2024)

ArXiv Pre-Print](https://arxiv.org/abs/2411.02577)


"We would like to see ourselves in the test:" The experiences of francophone African English learners in high-stakes English proficiency testing

Kadidja Koné, Paula Winke, & Matthew Gordon (2024)

Open Science Framework Preprints](https://doi.org/10.31219/osf.io/tsbf5)


Detecting LLM-Assisted Cheating on Open-Ended Writing Tasks on
Language Proficiency Tests

Chenhao Niu, Kevin Yancey, Ruidong Liu, Mirza Basim Baig, André Kenji Horie, and James Sharpnack (2024)

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track](https://aclanthology.org/2024.emnlp-industry.70.pdf)


A Generative AI-Driven Interactive Listening Assessment Task

Andrew Runge, Yigal Attali, Geoffrey T. LaFlair, Yena Park, and Jacqueline Church (2024)

Frontiers in Artificial Intelligence](https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2024.1474019/full)


The impact of task duration on the scoring of independent writing responses of adult L2-English writers

Ben Naismith, Yigal Attali, and Geoffrey T. LaFlair (2024)

Assessing Writing](https://www.sciencedirect.com/science/article/pii/S1075293524000886)


Duolingo English Test: Demographic and Score Properties of Test Takers (July 2023-June 2024)

Allison Michalowski, Ramsey Cardwell, Steven Nydick, and Ben Naismith (2024)

Duolingo Research Report](https://go.duolingo.com/demographic-score-2024)


Estimating Test-Retest Reliability in the Presence of Self-Selection Bias and Learning/Practice Effects

Will Belzak & J.R. Lockwood (2024)

Applied Psychological Measurement](https://journals.sagepub.com/doi/10.1177/01466216241284585)


BERT-IRT: Accelerating Item Piloting with BERT Embeddings and Explainable IRT Models

Kevin P. Yancey, Andrew Runge, Geoffrey LaFlair, and Phoebe Mulcaire

Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)](https://aclanthology.org/2024.bea-1.35.pdf)


Test preparation, test-taker attitudes, and high-stakes outcomes

Steven Nydick, Ramsey Cardwell, Jill Burstein, Duolingo

Presented at NCME 2024](https://duolingo-testcenter-prod.s3.amazonaws.com/media/resources/NCME+2024+eBoard+efficacy+study.pdf)


From Pen to Pixel: Rethinking English Language Proficiency Admissions Assessments in the Digital Era

Ramsey Cardwell, Ben Naismith, Jill Burstein, Steven Nydick, Sarah Goodwin, and Anthony Verardi (2024)

CALICO Journal](https://utppublishing.com/doi/10.1558/cj.27104)


Facilitating the Writing Process on the DET: The Interactive Writing Task

Sarah Goodwin, Mya Poe, Ramsey Cardwell, Andrew Runge, Yigal Attali, Phoebe Mulcaire, Kai-Ling Lo, and Geoffrey T. LaFlair (2024)

Duolingo Research Report DRR-24-02](https://duolingo-papers.s3.amazonaws.com/other/interactive-writing-whitepaper.pdf)


Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

Jiangang Hao, Alina A. von Davier, Victoria Yaneva, Susan Lottridge, Matthias von Davier, Deborah J. Harris

Educational Measurement: Issues and Practice](https://doi.org/10.1111/emip.12602)


Measuring Variability in Proctor Decision Making on High-Stakes Assessments: Improving Test Security in the Digital Age

William Belzak, JR Lockwood, & Yigal Attali (2024)

Educational Measurement: Issues and Practice](https://doi.org/10.1111/emip.12591)


Detecting careless cases in practice tests

Steven Nydick (2023)

Chinese/English Journal of Educational Measurement and Evaluation](https://doi.org/10.59863/LAVM1367)


Practical considerations when building concordances between English tests

Ramsey Cardwell, Steven Nydick, J.R. Lockwood, Alina A. von Davier (2023)

Language Testing](https://doi.org/10.1177/02655322231195027)


Fairness of using different English accents: The effect of shared L1s in listening tasks of the Duolingo English test

Okim Kang, Xun Yan, Masha Kostromitina, Ron Thomson, Talia Isaacs (2023)

Language Testing](https://doi.org/10.1177/02655322231179134)


Ensuring Fairness of Human- and AI-Generated Test Items

William C.M. Belzak, Ben Naismith, Jill Burstein (2023)

Artificial Intelligence in Education](https://link.springer.com/chapter/10.1007/978-3-031-36336-8_108)


The interactive reading task: Transformer-based automatic item generation

Yigal Attali, Andrew Runge, Geoffrey T. LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, Alina A. von Davier (2023)

Frontiers in Artificial Intelligence](https://doi.org/10.3389/frai.2022.903077)


Automated Evaluation of Written Discourse Coherence Using GPT-4

Ben Naismith, Phoebe Mulcaire, Jill Burstein (2023)

Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications](https://aclanthology.org/2023.bea-1.32.pdf)


Rating Short L2 Essays on the CEFR Scale with GPT-4

Kevin Yancey, Geoffrey T. LaFlair, Anthony Verardi, Jill Burstein (2023)

Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications](https://aclanthology.org/2023.bea-1.49.pdf)


Incorporating test security into the validity argument of a remotely-proctored English test

Ramsey Cardwell, Mancy Liao, Will Belzak, & Geoff LaFlair (2023)

Presented at LTRC 2023](https://go.duolingo.com/LTRC23Security)


Considering inter-test relationships in high-stakes admissions testing: The case of English

Ramsey Cardwell, Steven Nydick, J.R. Lockwood (2023)

Presented at the 8th International Conference of the Association of Language Testers in Europe ‘23](https://d23cwzsbkjbm45.cloudfront.net/media/resources/alte_23_cardwell.pdf)


The magic of machine learning: How to develop tests in multiple languages simultaneously and at scale

Sarah Goodwin, Lauren Bilsky, Phoebe Mulcaire, Burr Settles (2023)

Presented at the 8th International Conference of the Association of Language Testers in Europe ‘23](https://d23cwzsbkjbm45.cloudfront.net/media/resources/alte_23_goodwin.pdf)


Revisiting the language needs of students in English-medium universities

Ramsey Cardwell, Ben Naismith (2023)

Presented at the Biennial Conference of the British Association of Lecturers of English for Academic Purposes](https://d23cwzsbkjbm45.cloudfront.net/media/resources/baleap_23_cardwell.pdf)


Fairness Auditing in a Digital-First Learning & Assessment Ecosystem

Geoffrey T. LaFlair, Jill Burstein, Alina A. von Davier (2023)

Presented at the Annual Meeting of the National Council on Measurement in Education](https://go.duolingo.com/NCME23Fairness)


Sampling Considerations for Building Concordance Tables

Ramsey Cardwell, Steven Nydick, J.R. Lockwood (2023)

Presented at the Annual Meeting of the National Council on Measurement in Education](https://d23cwzsbkjbm45.cloudfront.net/media/resources/ncme_23_cardwell.pdf)


Plagiarism Detection Using Human-in-the-loop AI

Mancy Liao, Sinon Tan, Basim Baig (2023)

Presented at the Annual Meeting of the National Council on Measurement in Education](https://d23cwzsbkjbm45.cloudfront.net/media/resources/ncme_23_liao.pdf)


The regDIF R Package: Evaluating Complex Sources of Measurement Bias Using Regularized Differential Item Functioning

William C.M. Belzak (2023)

Structural Equation Modeling: A Multidisciplinary Journal](https://doi.org/10.1080/10705511.2023.2170235)


Human-in-the-loop automated item creation for complex items in language assessments

Geoffrey T. LaFlair, Yigal Attali, Andrew Runge, Kevin Yancey, Sarah Goodwin, Yena Park, Alina A. von Davier (2023)

Presented at the Annual Conference of the American Association for Applied Linguistics](https://d23cwzsbkjbm45.cloudfront.net/media/resources/aaal_23_LaFlair.pdf)


Finding time in language assessments: Maximizing measurement properties by unit of testing time

Sarah Goodwin, Geoffrey T. LaFlair, J.R. Lockwood, Steven Nydick, Alina A. von Davier (2023)

Presented at the Annual Conference of the American Association for Applied Linguistics](https://d23cwzsbkjbm45.cloudfront.net/media/resources/aaal_2023_goodwin.pdf)


Maintaining and Monitoring Quality of a Continuously Administered Digital Assessment

Mancy Liao, Yigal Attali, J.R. Lockwood, Alina von Davier (2022)

Frontiers in Education](https://doi.org/10.3389/feduc.2022.857496)


Duolingo English Test: Interactive Reading

Y. Park, G. LaFlair, Y. Attali, A. Runge, and S. Goodwin (2022)

Duolingo Research Report DRR-22-02](https://go.duolingo.com/interactive-reading)


Digital-First Learning and Assessment Systems for the 21st Century

Thomas Langenfeld, Jill Burstein, and Alina A. von Davier (2022)

Frontiers in Education](https://go.duolingo.com/Digital-First_LAS)


The Duolingo English Test: Psychometric considerations

G. Maris (2020)

Duolingo Research Report DRR-20-02](https://go.duolingo.com/drr-20-02)


Digital-first assessments: A Security Framework

Geoffrey T. LaFlair, Thomas Langenfeld, Basim Baig, André Kenji Horie, Yigal Attali, Alina A. von Davier (2022)

Journal of Computer Assisted Learning](https://go.duolingo.com/security_framework)


Analyzing the Duolingo English Test According to the AERA, APA, and NCME (2014) Standards

Maria Elena Oliver & Thomas E. Langenfeld (2021)](https://go.duolingo.com/Standards-Analysis-2021)


Improving Test Validity and Accessibility with Digital-First Assessments

Naomi Care & Bryan Maddox (2021)](https://go.duolingo.com/improving_test_validity)


The Multidimensionality of Measurement Bias in High-Stakes Testing: Using Machine Learning to Evaluate Complex Sources of Differential Item Functioning

William C. M. Belzak (2022)

Educational Measurement Issues and Practice](https://go.duolingo.com/multidimensionality)


Quality Assurance in Digital-First Assessments

Mancy Liao, Yigal Attali, Alina von Davier, J.R. Lockwood (2022)

Quantitative Psychology](https://doi.org/10.1007/978-3-031-04572-1_20)


Jump-Starting Item Parameters for Adaptive Language Tests

Arya D. McCarthy, Kevin P. Yancey, Geoffrey T. LaFlair, Jesse Egbert, Manqian Liao, and Burr Settles (2021)

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://go.duolingo.com/Jump-starting_item_parameters)


AQuAA: Analytics for Quality Assurance in Assessment (poster)

Manqian Liao, Yigal Attali, Alina A. von Davier (2022)

Proceedings of the 14th International Conference on Educational Data Mining](https://go.duolingo.com/AQuAA_poster)


AQuAA: Analytics for Quality Assurance in Assessment (paper)

Manqian Liao, Yigal Attali, Alina A. von Davier (2022)

Proceedings of the 14th International Conference on Educational Data Mining](https://go.duolingo.com/AQuAA_paper)


Duolingo English Test: Subscores

G. T. LaFlair (2020)

Duolingo Research Report DRR-20-03](https://go.duolingo.com/subscorewhitepaper)


Machine Learning Driven Language Assessment

B. Settles, G. T. LaFlair, & M. Hagiwara (2020)

Transactions of the Association for Computational Linguistics, 2020](https://doi.org/10.1162/tacl_a_00310)


The Duolingo English Test: Design, Validity, and Value

J. Brenzel and B. Settles (2017)

Duolingo Research Report](https://go.duolingo.com/detwhitepaper)


The Reliability of Duolingo English Test Scores

B. Settles (2016)

Duolingo Research Report DRR-16-02](https://go.duolingo.com/detreliability)


The Duolingo English Test and Academic English

L. Ishikawa, K. Hall, and B. Settles (2016)

Duolingo Research Report DRR-16-01](https://go.duolingo.com/detacademicenglish)


The Duolingo English Test and East Africa: Preliminary linking results with IELTS & CEFR

M. Bézy and B. Settles (2015)

Duolingo Research Report DRR-15-01](https://s3.amazonaws.com/duolingo-papers/reports/DRR-15-01.pdf)


Validity, Reliability, and Concordance of the Duolingo English Test

F. Ye (2014)

University of Pittsburgh Technical Report](https://go.duolingo.com/dettoefl)