Mining Process Data to Detect Aberrant Test Takers
Mining Process Data to Detect Aberrant Test Takers
Manqian Liao, Jeffrey Patton, Ray Yan, and Hong Jiao
ABSTRACT Item harvesters who memorize, record and share test items can jeopardize the validity and fairness of credentialing tests. Item harvesting behaviors are difficult to detect by existing statistical modeling approaches due to the absence of operational definitions and the idiosyncratic nature of human test-taking behaviors. To detect hard-to-define aberrant test-taking behaviors like item harvesting, we propose a data-mining approach that utilizes the process data to identify examinees whose test-taking processes deviate from the majority. This approach involves two steps: 1) learning archetypes of test-taking processes using the k-means clustering algorithm, and 2) flagging examinees whose behavioral patterns deviate from these archetypes for further investigation.
Introduction
Item harvesters can jeopardize the validity and fairness of credentialing tests. Unlike other abnormal testing behaviors, it is hard to delineate a behavioral pattern of item harvesting. The availability of process data allows for capturing subtle differences between aberrant test-takers and normal examinees. This study aims to develop a data-mining approach that utilizes process data to identify behavioral archetypes for the majority of examinees and detect those with deviant behaviors. Previous studies on cheating detection have used limited process data and strict parametric methods. Our proposed approach aims to capture additional types of aberrant test-takers.
Table of Major Types of Process Data Analyzed in This Study
| Process data type | Description | Example values |
|---|---|---|
| Response time | Time(in seconds) between starting to view an item and leaving for another | 69.5, 125.4 |
| Action type | The category of response (initial, answer change) | BTR, WTR, NC |
| Item presentation order | The viewing order of the item | 1, 2, 3 |
| Number of item visits | The number of times an item is visited by an examinee | 1, 2, 3 |
K-means Clustering
The k-means clustering algorithm is an unsupervised learning technique that divides data points into non-overlapping clusters. This study applies k-means clustering to detect abnormal test-taking behaviors, leveraging a wider variety of process data compared to existing methods. Specifically, we use clustering methods to learn archetypes of test-taking processes, followed by identifying examinees whose behaviors deviate.
Test-taking Process Representation
The test-taking process is represented as a sequence of actions that include response times, action types, and item presentation orders. The behaviors of examinees, analyzed using k-means clustering, lead to the identification of behavioral and examinee archetypes.
Learning Test-taking Process Archetypes
We divide the raw action sequences into segments and analyze them using k-means clustering to learn behavioral archetypes. The resulting clusters indicate the type and characteristics of the behaviors displayed by the examinees.
Detecting Outlying Test-taking Behaviors and Examinees
Outlier detection methods, based on k-means clustering, are implemented to identify both behavioral and person outliers. By establishing thresholds, we can flag individuals exhibiting abnormal behaviors for further investigation.
Conclusion
The proposed method offers an innovative approach to detecting aberrant test-taking behaviors using process data analyses. It serves as a powerful tool to discover unique differences in behavior patterns and to identify potential item harvesters. Further investigations are needed to validate the effectiveness of these findings and the method's applicability in various assessment contexts.