Jump-Starting Item Parameters for Adaptive Language Tests

Jump-Starting Item Parameters for Adaptive Language Tests

Authors

Arya D. McCarthy, Kevin P. Yancey, Geoffrey T. LaFlair, Jesse Egbert, Manqian Liao, and Burr Settles
1 Johns Hopkins University
2 Duolingo
3 Northern Arizona University

Abstract

(a) Easy for test-taker

A challenge in designing high-stakes language assessments is calibrating the test item difficulty, either a priori or from limited pilot test data. While prior work has addressed ‘cold start’ estimation of item difficulties without piloting, we devise a multi-task generalized linear model with BERT features to jump-start these estimates, rapidly improving their ability difficulty quality with as few as 500 test-takers and a small sample of item exposures (6 each) from a large item bank (4,000 items). Our joint model provides a principled way to compare test-taker proficiency, item difficulty, and language proficiency frameworks like the Common European Framework of Reference (CEFR). This also enables new item difficulty estimates without piloting them first, which in turn limits item exposure and thus enhances test security.

(b) Hard for test-taker

Moreover, introducing newly written items to replace or grow the item bank usually entails new rounds of piloting. In this paper, we present a principled method to jump-start new items without extensive piloting. A small amount of pilot data on older items (as few as 500 test-takers, each seeing 6 test items on average) is sufficient for calibrating new items’ difficulty and discrimination parameters.

1 Introduction

High-stakes language assessment demands high reliability, validity, and security. These goals are at odds with each other during the test design process. Large-scale pilot testing of new items to accurately measure their psychometric characteristics (e.g., difficulty and discrimination) risks that those items will be copied and leaked. Computer-adaptive tests, which more precisely score test-takers by selecting items of appropriate difficulty on-the-fly, exacerbate this conflict: their item banks must be large enough to cover all proficiency levels while ensuring the security of test items.

In this work, we fuse and generalize conventional Item Response Theory (IRT) with the work of Settles et al. (2020), remedying limitations of each. A single model estimates difficulty a priori to guide piloting; permits incremental learning from test-takers; and jump-starts difficulty estimates of newly written items, with similar quality as on observed items. We make the following contributions:

  1. We design and test a principled probabilistic model that incorporates arbitrary linguistic features of passage-based items. It measures item difficulty, test-taker ability, and CEFR level on a common logit scale.
  2. We show how BERT-derived passage embeddings facilitate strong generalization to new test items on a high-stakes English proficiency test.
  3. We provide linguistic validation of these difficulty estimates: they correlate strongly with lexico-grammatical features known to characterize reading complexity.

2 Background: Item Response Theory

Item response theory is the basis for most modern high-stakes standardized tests. It jointly assesses each person’s ability and the difficulty of each item in the item bank. The major distinction between IRT and its predecessors is that IRT assumes the item characteristics (e.g., difficulty) to be independent from the person characteristics (e.g., the person ability distribution).

The simplest such model is the Rasch model, a special case of logistic regression:

[ p(y=1|p,i)=\sigma(\theta_{p}-b_{i}) ]

where (\sigma) is the sigmoid function. Extensions of IRT allow polytomous responses and integrate temporal knowledge.

3 Approach and Formalism

We extend the 2PL model in two key ways. First, we decompose item difficulty into a weighted sum of features. Second, we integrate CEFR-labeled data for indirect supervision of passage difficulty within an ordinal-logistic regression multi-task model.

3.1 Student Modeling (TEST-TAKER)

To model our test-takers’ responses, we generalize 2PL into an LLTM by decomposing item discrimination and difficulty into multiple skills.

3.2 CEFR-labeled Dataset

We use 3,826 English passages automatically extracted and labeled with CEFR levels by two subject-matter experts. We hold out 10% of the passages for evaluation.

4.1 Data

We also use a collection of test-taker response data from the Duolingo English Test, extracting C-test item responses for this study. Each session contained test-taker responses to 5–8 distinct items drawn from an item bank of 4,151 unique C-test items.

4.2 Metrics

We measure performance on the CEFR data using the following metrics:

4.3 Baselines

The following models serve as baselines:

5 Results and Interpretation

Results indicate our joint model’s item parameters are better calibrated and more reliable than the estimates from SETTLES-ET-AL. Our model outperforms SETTLES-ET-AL on all five measures of test-taker modeling and achieves similar performance compared to the standard 2PL-IRT model, which requires significantly more item administrations.

6 Conclusion

We plan to show that BERT-LLTM's parameter estimates can be used as Bayesian priors for a 2PL-IRT model to achieve even more accurate results. Furthermore, we can expand to other item types and explore modeling the word-level difficulty of damaged words.

References

AERA, APA, and NCME. 2014. Standards for Educational and Psychological Testing. Agresti, A. 2010. Analysis of ordinal categorical data, volume 656. John Wiley & Sons. Settles, B., LaFlair, G. T., & Hagiwara, M. 2020. Machine learning–driven language assessment. Transactions of the Association for Computational Linguistics, 8:247–263.