The washback effect: How tests shape the way we study English

Dan Isbell is an Associate Professor in Second Language Studies at the University of Hawaiʻi at Mānoa whose research bridges language assessment and second language acquisition. He investigates the validity of language tests—both high-stakes and classroom-based—and how they can be used to support learning.

Goodhart’s Law asserts “when a measure becomes a target, it ceases to be a good measure”.

While this well-known adage originates in economics, it has been widely applied to many other areas of life, including language tests. But why?

High-stakes language tests can unlock doors—to education, employment, and migration opportunities. They also cost a bit of money, too. Naturally, if you plan to take a test, you’ll likely know what score you need (i.e., your target) and will want to prepare as best you can to reach it.

How people go about preparing for a test matters. If people just learn ‘tricks’ to get higher scores without actually getting better at the thing the test measures, like English, then the measure ceases to be a good measure! On the other hand, if a test motivates people to develop their English skills, that’s a good thing, and it makes Goodhart’s Law less of a concern.

From decades of research on language test preparation, we also know that the features of a test can influence how people prepare for it. As a newer test, and one with features that distinguish it from many other standardized tests of English (such as human-supported automated item generation, computer adaptive delivery, and at-home test taking), we don’t yet have a detailed understanding of how people go about preparing for the Duolingo English Test (DET).

In what follows, I will provide a bit more background on language test preparation and share some findings from a study on DET preparation my team and I conducted.

There’s more than one way to prepare for a test

Test preparation activities can be classified into three main types:

The types and intensity of test preparation people engage in is often linked to the uses and characteristics of the test and its stakes.

Tests influence the way we study English

Maybe you know someone who spent a lot of money and many late nights cramming anatomical vocabulary to take a medical exam, or perhaps you’ve heard about intensive, years-long preparation in public schools and private academies for university entrance exams in places like India and South Korea. Where the stakes are high, the test preparation can be intensive and often focused on Type 2 and Type 3 activities.

In the world of testing, this phenomenon is called washback: the impact a test can have on the learning process. Washback can be positive if it motivates people to learn something useful, or negative if it motivates people to spend a lot of time doing things that are not useful outside of the test. While it’s not the job of a test provider to teach you English (although some test providers have an app for that or offer courses), test providers do wish to minimize negative washback so that test takers and other stakeholders like teachers, parents, and administrators see the test as a complement to their learning efforts, rather than an obstacle or intrusion.

Connecting washback to test preparation types, it’s fine to see lots of Type 1 activity, and Type 2 activities in small doses. Type 3 activity clearly corresponds to negative washback, and is hopefully minimal, if not nonexistent.

So, how do people prepare for the DET?

It’s quite clear that people planning to take the DET commonly take the DET Practice Test at least once, but beyond that we don't have a very detailed understanding of how people go about preparing for the test.

With a large team of researchers affiliated with the University of Hawaiʻi at Mānoa, I’ve been interviewing DET test takers and poring over test preparation resources to better understand the range of activities people engage in to get ready for the DET.

Our research has included students currently studying at universities in the US and Canada, as well as popular English-language YouTube channels and online courses. We also decided to include a focus on the East Asia region, where exam preparation is extremely prevalent and where a large portion of DET test takers are located (see the DET team’s report on scores and demographics).

Thanks to our research team’s language skills, we’ve been able to interview test takers, test prep coaches, and materials developers—39 people in total—and analyze test preparation materials produced in English, (Mandarin) Chinese, Japanese, and Korean, reviewing over 160 sources in total!

The verdict on washback

Our findings suggest that getting familiar with the test by taking the practice test, practicing specific test tasks, and reading/watching information about the tests (in other words, Type 2 preparation) is most commonly featured in preparation materials and mentioned by interviewees.

Doing things that develop general English skills (Type 1) were less commonly reported by interviewees or featured in test preparation materials. Focusing on tricks to get a better score (Type 3) was least common.

In a bit more detail, we also found some interesting links between the design and administration of the DET, and how people prepare for it:

In the materials we analyzed and interviews we conducted, we saw evidence for both positive and negative washback through Type 1 and Type 3 preparation, respectively. We also saw lots of Type 2 preparation focused on test familiarization, which is potentially neutral. But it’s hard to generalize the washback effect of a test to every test taker.

Some people we interviewed tried the official practice test, watched a video or two about how to take the test, and then took their official DET in the same afternoon. This might be the ideal scenario for test preparation, even though there was no activity leading to increased English skills! On the other hand, some people prepared intensively for several weeks or longer, mixing Type 2 and Type 3 preparation.

Keeping test preparation positive

So what should you do? While everyone’s test taking journey will look a little different, I recommend aiming for a positive preparation experience: get familiar with the test, and focus on developing genuine English skills as needed to achieve your goals.

First, a large body of research on standardized tests suggests that taking a practice test is the single best thing you can do to ensure you perform your best on an official test, so take advantage of the official DET Practice Test, and watch a few videos on the DET YouTube channel to make sure you understand how to set up your room and computer for a secure official test.

If your typing speed in English using a keyboard is on the slower side, doing some typing practice wouldn’t be a bad idea (plus, this is something likely to be useful in real life!). If your target score is a bit above your current abilities, you’ll want to make a longer-term study plan that includes things like vocabulary and grammar study, but I also recommend practicing using English—something that was unfortunately mentioned a bit less commonly than studying vocabulary and grammar in our data.

Preparing for the DET can include fun activities like watching movies, reading the news, participating in a language exchange, and posting on social media. All of these will help grow your language skills!