Consequential Validity Evidence of Yes-No Angoff Standard Setting in a Pre-Clinical Medical School Curriculum

doi:10.21203/rs.3.rs-4889026/v1

Download PDF

Research Article

Consequential Validity Evidence of Yes-No Angoff Standard Setting in a Pre-Clinical Medical School Curriculum

https://doi.org/10.21203/rs.3.rs-4889026/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

Standard setting is consequential to student outcomes for defining sufficient mastery of knowledge and thus accurately classifying medical students as competent (or not) for advancing through a pre-clinical curriculum. In particular, for multiple-choice medical knowledge exams the Yes-No Angoff method for standard setting yields consequential cut-off scores based on faculty-experts’ item-level judgments of students possessing borderline but sufficient competence required for demonstrating sufficient mastery. Adapting a construct validity framework and defining consequential academic success in terms of unimpeded progress, we investigated consequential validity evidence of passing standards derived from the Yes-No Angoff method.

Methods

We analyzed academic success for four pre-clinical semesters across three student cohorts. First, we identified passing standards for pre-clinical courses using the Yes-No Angoff method. Then, we applied binary logistic regression and receiver-operator characteristic (ROC) analyses with area under the curve (AUC) to evaluate passing standards. For binary outcomes, we defined academic success in terms of unimpeded progress through the curriculum and students’ first-attempt passage of the United States Medical Licensing Examination (USMLE) Step 1. Model predictors included Yes-No Angoff passing standards, Medical College Admissions test scores, and grade point averages for math and science courses.

Results

ROC analyses showed a low but acceptable area under the curve for a single semester in one cohort and excellent or outstanding AUCs for the remaining 11 semesters. With rates ranging between 89% and 96%, overall classification accuracy for predicting academic success was adequate for all pre-clinical semesters across all cohorts.

Conclusions

Affirming that Yes-No Angoff standards accurately predicted academic success, study results indicated consequential validity evidence for our school’s passing standards in a pre-clinical medical curriculum.

medical education

standard setting

Yes-No Angoff

consequential validity

Undergraduate medical education typically requires a pre-clinical curriculum that focuses on the basic science knowledge and professional performance competencies students will need for clinical education and postgraduate residency training. For assessing knowledge in the pre-clinical curriculum, medical schools set passing standards for courses that must be achieved in order for students to advance. A school may employ any number of methods for deriving cut-off passing scores for exams and passing standards for courses. Selected standard setting methods include historic precedent, Angoff, Yes-No Angoff, Bookmarking, Contrasting Groups, Ebel, and Hofstee.^1,2

Adapted from Angoff standard setting, the Yes-No Angoff method is criterion-based and thus relies on the expert judgements of raters. With its relative ease of use and efficiency in administration, faculty experts rate exam questions based on subjective predictions of the item-level performance of “borderline” students performing at a level sufficient to pass and move on in the curriculum.² Construct validity defines how well a test measures the concept it was designed to evaluate. Construct validity becomes important when assessing a concept that cannot be directly measured, such as competence, and multiple measurable indicators, such as test items, are needed.³

Ultimately, these item-level predictions of raters yield passing standards that define the borderline level of competence that is yet sufficient for progressing through a curriculum and adequate for attaining a terminal degree. Any given method, systematically and rigorously applied, can yield fair and defensible standards.¹ And yet, for any given method, standards are only as fair and defensible as the evidence for validity warrants. With this imperative in mind, we investigated the consequential validity of passing standards that were derived from the Yes-No Angoff method.^3,4,5,6,7,8

This study was reviewed by the Mercer University School of Medicine Institutional Review Board and deemed not to involve health care intervention, and need for consent waived, and ethics approval granted, with reference number H2212295.

Our school’s pre-clinical courses are organized in terms of four semester long blocks. For each block we assessed for students’ mastery of biomedical science with locally developed multiple choice question (MCQ) exams, setting standards per the Yes-No Angoff method. Our data included standard setting outcomes for three cohorts of medical students in 2017, 2018, and 2019, N = 126, 112, and 116 for each cohort. Over these years there were 24 exams in block 1 and 22 in block 2 with a total of 3025 items in the first pre-clinical year. There were 18 exams in block 3 and 24 in block 4 with 2775 items in the second pre-clinical year. The numbers of Yes-No Angoff raters typically ranged between 12 and 18 faculty members per exam.

In our application of the Yes-No Angoff method, faculty members reviewed each MCQ exam item, and for that item answered the question, “Will the borderline student answer this question correctly?” For each item, faculty rated 1 for Yes or 0 for No. From these binary data, we summed the means of item ratings to determine cut-off scores for each exam. Then, we totaled the sum of cut-off scores to calculate pre-clinical passing standards for each block. This system for determining block passing standards was compensatory such that better performance on some exams compensated for poor performance on others. Table 1 illustrates an item × rater matrix for demonstrating how item means were calculated and then summed to determine the cut-off scores for exams. For example, the sum of item means and final cut-off score for this 10-item exam is 6.714.

Table 1

Example of Yes-No Faculty Angoff Ratings by 7 faculty for 10 Exam Items
Raters	1	2	3	4	5	6	7	Mean
1	1	0	1	1	1	0	1	0.714
2	0	1	0	1	1	0	1	0.571
3	0	1	1	0	1	0	1	0.571
4	0	0	1	1	0	1	1	0.571
5	1	1	1	1	1	1	1	1.000
6	1	1	1	1	1	1	1	1.000
7	1	0	1	1	0	1	1	0.714
8	0	0	0	1	1	1	0	0.429
9	1	0	0	0	1	1	1	0.571
10	0	1	1	0	1	0	1	0.571
Item Totals	5	5	7	7	8	6	9	6.714

We analyzed binary logistic regression results and receiver-operator characteristic curves (ROC) to evaluate consequential validity of passing standards.^9,10 Our independent variable was Academic Success, defined as students’ unimpeded progress through the pre-clinical curriculum plus their first-attempt passage of USMLE Step 1. The principle predictor in our models was Faculty Standard, which represented the dichotomized outcomes of students scoring at or above faculty passing standards versus those who scored below. Model predictors also included Medical College Admissions Test scores (MCAT) and pre-matriculating grade-point-averages for biology, chemistry, physics, and math courses (BCPM-GPA).¹¹

First, we calculated predicted probabilities and overall classification accuracy for Academic Success. Then, we plotted these predicted probabilities against observed outcomes for Academic Success in order to construct ROC-curves. Inferring validity evidence from ROC-curves, we applied conventional guidelines for evaluating area under the curve (AUC): AUC of 0.5 to 0.59 is poor, 0.6 to 0.69 is fair, 0.7 to 0.8 is good, 0.8 to 0.9 is excellent, and more than 0.9 is considered outstanding.¹²

In plotting predicted probabilities against actual outcomes, ROC-curves illustrate simultaneously the true and false positive rates of students who should and should not have passed, showing the probability of success or failure across all probability thresholds between 0 and 1. For each pre-clinical block, Fig. 1 illustrates the ROC curves with the Yes-No Angoff method for predicting academic success, with all ROC-curves per cohort bending toward the upper-left corner. ROC curves closer to the upper-left corner indicate better evidence for validity. Thus, with MCAT, BCPM-GPA, and Faculty Standard as predictors, these curves visually demonstrate the goodness of fit of our models for predicting Academic Success.

Complementing our study’s ROC-curves, Table 2 documents the AUC for all pre-clinical blocks and cohorts. The 2018 cohort’s Block 3 AUC was somewhat low but acceptable at 0.7941. All other models were excellent or outstanding with AUC results ranging between 0.8176 and 0.9238.

Table 2

Area Under the Curve for Block Totals
	Block 1	Block 2	Block 3	Block 4
2017 Cohort	0.8697	0.9153	0.8962	0.9195
2018 Cohort	0.8176	0.8961	0.7941	0.9069
2019 Cohort	0.8992	0.9238	0.9096	0.9111
Note. AUC = 0.7 to 0.799 is acceptable; AUC = 0.8 to 0.899 is excellent; AUC = 0.9 + is outstanding.

As referenced above, we defined Academic Success in terms of students’ unimpeded progress through the pre-clinical curriculum and their first-attempt passage of the USMLE Step 1. Across all cohorts, the actual rates of unimpeded progression through pre-clinical blocks were 90% for Blocks 1, 2, and 3, and 95% for Block 4. Our school’s first-time pass rate for USMLE Step 1 across all cohorts was 91%, compared with 94 to 96% for U.S. and Canadian medical school examinees in 2017-2019.¹³ The rate of Academic Success, as defined for binary logistic regression, was 79%. As a measure of overall accuracy, the rates of correct classification in our models ranged between 89% and 96%.

Assessment requires validity evidence that affirms academic standards, gives meaning to test scores, and informs high stakes decisions for advancing students through a curriculum.^3,5,6,7,8 In our study, “consequential validity” was defined in terms of the binary outcome, Academic Success. This outcome and our method for investigating validity are based on the work of Dunleavy et al who defined academic success in terms of “unimpeded progress” and investigated predictive validity with binary logistic regression and ROC-curve analyses.⁴ In this way of thinking about validity: (1) the probabilities for achieving Academic Success were predicated, in large part, on faculty-set passing standards; and (2) AUC indicated goodness of fit and thus the quality of consequential validity evidence. Per our study’s method, AUC findings indicated adequate goodness of fit in our predictive models. Moreover, rates of classification accuracy affirmed the consequential outcomes of students progressing through the curriculum unimpeded and passing Step 1 on their first try.

Significantly, validity ultimately concerns both the consequences of test use and the accuracy of inferences made about psychological constructs.^4,8,14 Thus, in addition to validating standard setting outcomes, our study’s results provided validity evidence for the foundational psychological construct that defines the Yes-No Angoff procedure, namely “borderline competence.”^1,2 Accordingly, our results pointed to consequential validity evidence that most students had developed (or not) the necessary competence in biomedical science. However, our models were not always helpful in predicting the likelihood that students who failed truly lacked the necessary competence for medical knowledge. Our school anticipated this limitation when it adopted the Yes-No Angoff procedure and therefore developed a protocol for adjusting faculty standards with an approach suggested by Camara et al.¹⁵ As such, we addressed the possible classification error of unwarranted impeded progress by adjusting cut-off scores downward according to our calculations for standard error of measurement (SEM).^15,16,17 Concurrently, we adjusted faculty standards upward per SEM to address the other kind of classification error: that of passing incompetent students. For this kind of classification error, upward adjustments also helped us identify students who needed academic support. Ultimately, we see our application of SEM as sound protocol in standard setting that mitigates undesirable consequences while supporting the appropriate use of academic standards.⁸

It should be noted that the borderline student is not an honors student or even an average student; but neither is the borderline student a failing student. The borderline competent student may struggle to pass at a given point in the curriculum, but having once passed, will have demonstrated the sufficient competence required for advancing through the pre-clinical curriculum. Faculty development aimed at honing expert raters’ shared understandings of borderline competence is foundational for supporting accurate inferences.^18,19 Moreover, we hold that validation of passing standards should be continuous in order to reflect the dynamic nature of testing for locally-developed assessments. Accordingly, we advocate for annual reviews of predictive models and classification statistics that affirm the competence of high achievers but also help schools of medicine identify students who need additional academic support. As regards our school’s dynamic application of the Yes-No Angoff method across multiple cohorts we observed a profoundly important value-added to teaching and learning: the systematic review of exams, item by item, was a necessary task fulfilled by multiple faculty colleagues. In reviewing items, faculty provided feedback on the accuracy of test items, their formatting and relevance to medical practice, and their alignment with curriculum goals and objectives. These reviews supported the quality improvement of new MCQ-items developed for new knowledge content in our pre-clinical curriculum.

Certainly, our cohorts included a number of students who were not coded for Academic Success according to our study’s stringent rules for defining the binary outcome. And yet, most of our students who failed to achieve so-called Academic Success did in fact achieve ultimate success after remediation in the pre-clinical curriculum or after their passage of Step 1 on a second try. We would note that all of our matriculated students have gone through a rigorous evaluation process for selection and that our school’s curriculum is stepwise such that faculty do not expect students to have achieved full expertise all at once. It should also be noted that the typical medical school incorporates six competencies into a curriculum, not just medical knowledge. Thus, an MD graduate must also demonstrate competency in interpersonal and communication skills, professional behavior, patient care, systems-based practice, and practice-based learning and improvement.²⁰ With this broader view of the knowledge, skills, and attitudes students need for a successful professional career, the importance of consequential validity evidence comes into sharper focus as schools of medicine assess for more than just a single competency. Ultimately, validity studies for all competencies are required for affirming students’ probable success in graduate medical education and their career readiness for the entry-level responsibilities of first-year residents.

Out study showed that the Yes-No Angoff method was adequate for predicting academic success and therefore indicative of consequential validity. Moreover, the method provided a means for setting valid standards with substantial faculty input. This input supported test development, contributed to MCQ-item accuracy for inferring mastery of knowledge, and improved the clinical relevance of our exams. Significantly, our school’s systematic application of the Yes-No Angoff method aided in identifying borderline performing students in need of academic support. Future studies might consider the range of values in binary logistic cut-points for a more comprehensive discussion of positive and negative predictive values and the tradeoff associated with varying cut-point thresholds.

ROC

AUC

USMLE

MCQ

MCAT

BCPM-GPA

SEM

Ethics approval and consent to participate: This study was reviewed by the Mercer University School of Medicine Institutional Review Board and deemed not to involve health care intervention, and need for consent waived, and ethics approval granted, with reference number H2212295.

Consent for publication: Not applicable.

Availability of data and materials: The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests: The authors declare that they have no competing interests.

Funding: None.

Authors’ contributions: All authors contributed equally to the analysis of the data, the preparation of the manuscript, and read and approved the final manuscript.

Acknowledgements: Not applicable.

Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med. 2006;18(1):50–7. https://doi.org/10.1207/s15328015tlm1801_11.
Yudkowsky R, Downing SM, Wirth S. Simpler standards for local performance examinations: The Yes/No Angoff and Whole Test Ebel. Teach Learn Med. 2008;20(3):212–7. https://doi.org/10.1080/10401330802199450.
Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. ETS Res Rep Ser. 1994;1994(2):i–28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x.
Dunleavy DM, Kroopnick MH, Dowd KW, Searcy CA, Zhao X. The predictive validity of the MCAT exam in relation to academic performance through medical school: A national cohort study of 2001–2004 matriculants. Acad Med. 2013;88(5):666–71. https://doi.org/10.1097/ACM.0b013e3182864299.
Kane M. Validating the performance standards associated with passing scores. Rev Educ Res. 1994;64:425–61. https://www.jstor.org/stable/1170678.
Kane MT. Validating the Interpretations and Uses of Test Scores. J Educ Meas. 2013;50(1):1–73. https://doi.org/10.1111/jedm.12000.
Hambleton RK. Setting performance standards on educational assessments and criteria for evaluating the process. In: Cizek GJ, editor. Setting Performance Standards: Theory and Applications. New York: Routledge; 2001.
Bachman LF. Building and supporting a case for test use. Lang Assess Q. 2005;2(1):1–34. https://doi.org/10.1207/s15434311laq0201_1.
Berwick V, Cheek L, Ball J. Statistics review 14: Logistic regression. Crit Care. 2005;9(1):112–8. https://doi.org/10.1186/cc3045.
Eng J. Receiver operating characteristic analysis: A primer. Acad Radiol. 2005;12(7):909–16. https://doi.org/10.1016/j.acra.2005.04.005.
Kleshinski J, Khuder SA, Shapiro JI, Gold JP. Impact of preadmission variables on USMLE step 1 and step 2 performance. Adv Health Sci Educ. 2009;14:69–78. https://doi.org/10.1007/s10459-007-9087-x.
Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thoracic Oncology. 2010;5(9):1315–1316. https://doi.org10.1097/JTO.0b013e3181ec173d.
USMLE Performance Data. https://www.usmle.org/performance-data. Accessed 9 August 2024.
Loevinger J. Objective tests as instruments of psychological theory. Psychol Rep. 1957;3(3):635–94. https://doi.org/10.2466/pr0.1957.3.3.635.
Camara WJ, Allen JM, Moore JL. Empirically based college- and career-readiness cut scores and performance standards. In: McClarty KL, Mattern KD, Gaertner MN, editors. Preparing Students for College and Careers. 1st ed. New York, NY: Routledge; 2018. pp. 70–81.
Hays R, Gupta TS, Veitch J. The practical value of the standard error of measurement in borderline pass/fail decisions. Med Educ. 2008;42(8):810–5. https://doi.org/10.1111/j.1365-2923.2008.03103.x.
Biddle RE. How to set cutoff scores for knowledge tests used in promotion, training, certification, and licensing. Public Personnel Manage. 1993;229(1):63–79. https://doi.org/10.1177/009102609302200105.
Clauser JC, Clauser BE, Hambleton RK. Increasing the validity of Angoff standards through analysis of judge-level internal consistency. Appl Measur Educ. 2014;27:19–30. https://doi.org/10.1080/08957347.2013.853071.
Tannenbaum RJ, Kannan P. Consistency of Angoff-based standard setting judgments: Are item judgments and passing scores replicable across different panels of experts? Educational Assess. 2015;20:66–78. https://doi.org/10.1080/10627197.2015.997619.
Coalition for Physician Accountability. Consensus Statement on a Framework for Professional Competence by the Coalition for Physician Accountability. https://physicianaccountability.org/wp-content/uploads/2020/05/Coalition-Competencies-Consensus-Statement-FINAL.pdf. Updated. 2014. Accessed 22 May 2024.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
20 Aug, 2024
Editor assigned by journal
19 Aug, 2024
Submission checks completed at journal
19 Aug, 2024
First submitted to journal
09 Aug, 2024

You are reading this latest preprint version

Consequential Validity Evidence of Yes-No Angoff Standard Setting in a Pre-Clinical Medical School Curriculum

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1