Duolingo English Test

Research Governance Committee

Overall objective

The Duolingo English Test (DET) will maintain a Research Governance Committee (RGC) in order to help administer our Competitive Research Grant cycle, Commissioned Research, and other External Research Collaborations.

The overall purpose is for the RGC to provide advice to DET leadership regarding the scientific merit of the Grants issued and the proposals received from the public. The RGC will serve to uphold research integrity for DET-funded research. To do this, they will follow research conduct guidelines from the assessment community.

Specific objectives

The RGC will have the following specific objectives:

1. Uphold high standards of rigor and integrity in all aspects of research:

a. Ensure that research is conducted according to appropriate ethical, legal, and professional frameworks, obligations, and standards

b. Use transparent, timely, and fair processes to address allegations of research misconduct if they arise

2. Maintain alignment with DET research goals, updates, and new versions of the test

3. Facilitate successful communication between DET R&D leadership and external research collaborators

Responsibilities

The RGC will have responsibilities in three areas:

1. Call for research proposals

• Advise about research topics to support the validity of the DET

• Help determine reasonable project parameters, such as budgets and timelines, in collaboration with DET leadership

• Evaluate risk in terms of inadvertent proprietary information exposure and study impact

• Review & uphold the rubric used to evaluate research proposals (see rubric below)

2. Reviewing research proposals

• Provide fair proposal review

• Incorporate a double-anonymized peer review process

• Use standardized rubric criteria for reviewer evaluations

• Provide constructive feedback that can be shared with proposal authors on request

• Consider the appropriateness of the proposed data samples, instruments, and methods

• Ensure ethical treatment of study participants

• Proposals should be vetted by ethical review boards at the investigator’s institution to ensure that the study:

• upholds the rights, safety and well-being of study participants

• guarantees appropriate ethical and other approvals are obtained in line with project plans

• Evaluate the overall meaningfulness of the proposed study to DET stakeholders, including members of academic, institutional, and governmental communities

3. Reviewing research reports of funded projects

• Comment on the fidelity of methods used, reporting of findings, and inferences drawn from said findings

• Comment on interim and final study reports in terms of meaningfulness, completeness, and soundness of interim and final findings

• Make recommendations to the Chief of Assessment to approve final reports

• Comment on the alignment with the DET research goals

• Provide advice regarding the viability of external dissemination of results (such as peer-reviewed conference presentations and journal article publications) and make recommendations to the Chief of Assessment to approve or reject internal (for example, blogs, white papers) or external dissemination (conference presentations, journal article publications, etc.)

Committee structure

The Committee will be composed of the following members:

• Three (3) external members (from the academic community), each of whom may serve for a 3-year term with one optional 3-year renewal. External members cannot have a current affiliation with Duolingo (such as serving on another advisory board)

• Two (2) members of the DET Validity, Efficacy and Outreach team

• One (1) member of the DET Institutional Engagement team

Term

The Committee will be appointed for a fixed term of three years. They will meet at least two times a year, at least once online and once in person in Pittsburgh with other members of the RGC. They may present their work to the DET’s Technical Advisory Board in one of the virtual meetings once per year.

Transparency

To build public trust through transparency, the RGC will make recommendations on how to make research available to a wide and relevant stakeholder audience. Examples include, but are not limited to, recommendations about DET research presentation and publication venues, public data sets, and research methods to allow for replication.

Honoraria for external members

External committee members will be provided with an honorarium for their participation. This honorarium covers the time and preparation for meetings.

Conflict of Interest for external members

• External members will disclose any conflict of interest they may have (advisory membership of competing institutions).

• External members will need to provide a declaration that they are permitted to receive honoraria from Duolingo for their work on the RGC.

Competitive Research Grant Proposal Rubric

Relevance of proposal to topics listed in the Call for Proposals [5-point scale]

• 5 = Highly relevant with no reservations

• 4 = Relevant with minor reservations

• 3 = Somewhat relevant

• 2 = Some relevant parts and some irrelevant parts

• 1 = Marginally relevant

• 0 = No relevance

Clarity of rationale [5-point scale] + Completeness of literature review [5-point scale]

• 5 = Very clear questions and complete literature review with no reservations

• 4 = Clear questions and literature review with minor reservations

• 3 = Somewhat clear questions and literature review with reservations

• 2 = Some clear and some unclear questions and literature with major reservations

• 1 = Unclear questions and incomplete literature review

• 0 = No research questions and literature review

Significance and impact of proposed research to the DET and language assessment research community [5-point scale]

• 5 = The proposed research is highly significant and highly impactful to the DET and to the language research community

• 4 = The proposed research is significant and impactful to the DET and to the language research community

• 3 = The proposed research is marginally significant and marginally impactful to the DET and to the language research community

• 2 = The proposed research is minimally significant and minimally impactful to the DET and to the language research community

• 1 = The proposed research is insignificant and unimpactful to the DET and to the language research community

• 0 = The proposed research is not significant and not impactful to the DET and to the language research community

Appropriateness of study design and methodology [5-point scale]

• 5 = Very clear methods with no reservations

• 4 = Clear methods with minor reservations

• 3 = Somewhat clear methods

• 2 = Some clear and some unclear methods

• 1 = Unclear methods

• 0 = inappropriate methods

Researcher’s qualifications and background match to proposal needs [3-point scale]

• 3 = Researcher’s background demonstrates ample qualification to successfully deliver on the research as described, demonstrated by a clear record of peer-reviewed publications and past studies of a relevant nature

• 2 = Researcher’s qualifications indicate a strong capacity to deliver on the research as described, evidenced by a consistent record of peer-reviewed publications and some relevant past studies

• 1 = Researcher’s qualifications show limited potential to conduct the research, supported by a limited number of peer-reviewed publications on unrelated topics

• 0 = There is no ground to assess the researcher’s qualifications and/or background

Feasibility in terms of time, cost, and project logistics [3-point scale]

• 3 = Project is not only practically feasible but also requires minimal or no investment of time & energy from DET personnel; project’s cost and timeline are reasonable for the work and clearly justified by the proposal

• 2 = Project is practically feasible and requires moderate but worthwhile investment of time & energy from DET personnel; project’s cost and timeline are mostly reasonable and justified by the proposal

• 1 = Project’s practical feasibility is questionable, requires heavy investment of time & energy from DET personnel; project’s cost and timeline are barely justified by the proposal

• 0 = Project is technically infeasible given current limitations

Understanding of the current DET [3-point scale]

• 3 = Author shows an excellent understanding of the DET, demonstrated through their description of the DET, inclusion of cited DET research, and up-to-date framing of the DET

• 2 = Author shows a mostly acceptable understanding of the DET, but some aspect(s) of their description of the DET, inclusion of cited DET research, and framing of the DET may be lacking

• 1 = Author shows a somewhat acceptable understanding of the DET, but describes the DET in a way that is out of date, cites irrelevant research, or frames the DET in an outdated lens

• 0 = Author shows an unacceptable understanding of the DET, demonstrated through a description of the DET that contains obvious inaccuracies, few or no citations of current DET research, and misleading or inaccurate framing of the DET

Duolingo English Test

Competitive Research Grants, 2023-2024

Proposal Evaluation Rubric