Minimum Accepted Competency Examination: Test Item Analysis

doi:10.21203/rs.3.rs-956107/v1

Download PDF

Research Article

Minimum Accepted Competency Examination: Test Item Analysis

https://doi.org/10.21203/rs.3.rs-956107/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

The minimum accepted competency (MAC) examination in paediatrics was a set of MCQ’s designed to test undergraduate medical students on the most basic, ‘must know’ knowledge as determined by non-faculty paediatric clinicians. The faculty standard set the examination at a relatively difficult 41.2% with only two-thirds of students attaining this score (even though 96% of the same students passed their university paediatric examination). We will now describe the psychometric properties of the examination in a bid to explain these results and provide specific suggestions for future practice

Methods

The methodology for designing the content, standard setting and administration of the MAC examination has already been described. The examination was set by three groups of undergraduate medical students and paediatric trainee doctors. Test item analysis was performed using difficulty index, discrimination index and Cronbach’s alpha.

Results

Test item difficulty between each group was positively correlated. Correlation of item difficulty with the standard set for each item showed a statistically significant positive relationship. However, for 10 of the items, the mean score did not even reach two standard deviations below the standard set by the faculty. Medical students outperformed the trainee doctors on three items. 18 of 30 items achieved a discrimination index >0.2. Cronbach’s alpha ranged from 0.22-0.59.

Conclusion

Questions on respiratory paediatrics and seizures were answered poorly. Despite these topics being major topics within the curriculum, it is possible that the clinical emphasis in these non-faculty derived questions was juxtaposed with the factual recall often required for university examinations. Interestingly, for the questions on which the students performed better than the doctors, the questions were quite specific, and the answer found directly in the curriculum. This reflects how students learn with limited clinical experience and is a good example of assessment being a key driver of learning.

The concept of a MAC examination to test basic medical knowledge is feasible and the study presented is an encouraging first step towards this method of assessment. This method is readily transferable to other medical (and allied health) specialties.

Educational Psychology

Medical education

Assessment

Standard setting

Multiple choice questions

Competence

This study investigates whether undergraduate training adequately prepares doctors with the required knowledge to practice paediatrics at the early senior house officer (SHO) level. The hypothesis is that there are relevant areas of the paediatric undergraduate curriculum, which are either missed, not highlighted or not incentivised which results in students successfully passing undergraduate assessment and then entering the workplace with crucial knowledge deficits (1).

To that end, we designed a novel approach to assessment in which question content was provided by non-faculty paediatric clinicians with the remit of ‘must know basic knowledge for a doctor starting work in paediatrics,’ (i.e., SHO). This examination was named, Minimum Accepted Competency (MAC) and the methodology and gross examination results have previously been described (2). The logic behind this approach is that we wanted to capture a finite amount of the most relevant crucial knowledge that students would be assessed in, thereby equipping them with the knowledge to tackle most clinical tasks early in in their career and provide a solid platform on which to build. The expectation was that, as the questions were meant to be ‘easy’, that the passing score for the examination would then be appropriately high (i.e., candidate must get most of the questions correct to pass the examination), thus would then ensure that no student graduated medical school and potentially start work in paediatrics without a firm grasp of this finite level of knowledge.

Following standard setting, the passing score for the MAC examination was calculated at 41.2%. The average score achieved by undergraduate students was 46% with only two thirds reaching the pass mark, whereas in the university paediatric examination, 96% of students passed. The average score achieved by postgraduate SHO’s was 64% (2). Therefore, we concluded that there must be a gap in the expectation of knowledge between non-faculty paediatricians and the academic paediatric faculty. Overall candidates performed poorly and therefore did not have the required knowledge to succeed in this type of assessment.

We will now delve deeper into the content of the MAC examination and describe the psychometric results (as opposed to simply the gross score obtained by each candidate) in a bid to explain why candidates performed how they did and provide more specific suggestions for future practice.

The methodology for designing the content, standard setting and administering the MAC examination has already been described (2). The MAC examination was sat by four different groups; three undergraduate groups (Year 1 Royal College of Surgeons in Ireland (RCSI)), Year 2 RCSI and Queen’s University Belfast (QUB)) and one postgraduate group (basic specialist trainees (BST) i.e. SHOs). Once the results were collected, we performed test item analysis, difficulty index [DD], discrimination index [DI] and Cronbach’s Alpha. Test item analysis is used to identify which questions were most difficult and whether this is consistent between the studied groups.

Item difficulty (DD) is measured by the proportion of candidates who correctly answered a given test item. Therefore, a test item, which was difficult, is flagged by a low percentage of candidates giving a correct response to that item (3). Some authors have suggested that a test item should attempt to have a difficulty index (DD) in the range between 0.3-0.7 (4). We used Spearman’s rank to correlate item difficulty between the different groups who sat the examination. We also correlated item difficulty with that same item’s standard set using Pearson correlation coefficient.

The discrimination index (DI) is the point-biserial correlation coefficient between the test item and the mean score. This examines whether a test item is able to distinguish between a student of high ability and those with low ability (5)(6). A DI < 0 (i.e., a negative value) would mean that the best students got this question wrong more often than the worst students. DI= 0 would mean it was a poor discriminator between good and bad students. A test item, which achieves DI >0.2 is deemed ‘good’ and therefore a re-useable question for future sittings of that examination (4).

The Cronbach’s alpha is a measure of internal consistency. It is a description of the extent to which all the items in a test measure the same concept and is, therefore, a way of expressing the inter-relatedness of the items within the test (7).

In the absence of any further guideline, we used the five most difficult questions from the MAC examination as examples to consider if there is a pattern between the studied groups' performance. We also used the arbitrary measure of two standard deviations below the faculty-derived standard set for each question as a marker of particularly poor performance if not met.

We analysed in more detail the items which we have identified as particularly difficult for the candidates and try to explain why this might have been the case. We indicate whether the answer to the question could have been found in the university paediatric curriculum (and therefore arguably should have been prepared for by the undergraduate students who sat the MAC exam).

A copy of the MAC examination is available in the appendix.

When the test item difficulty is ranked within each group, a Spearman’s rank correlation indicates a positive correlation between Year 1 RCSI and Year 2 RCSI 0.80 (p<0.01), Year 1 RCSI and SHO 0.73 (P<0.01)) and Year 2 and SHO 0.66 (P<0.01) (see figure1).

Correlations of item difficulty with standard set mark for each item on the MAC exam show a statistically significant positive relationship for both undergraduate groups. For Year 1 RCSI, the Pearson correlation coefficient was 0.66 (p<0.01), for Year 2 RCSI, it was 0.62 (p<0.01). Therefore, overall, the most difficult questions as set by the faculty proved to be most difficult for the examinees also. However, when we look in more detail, for items 1,3,6,7,8,10,12, 15,18 and 28, the mean score for one or more of the study groups did not even reach 2 standard deviations below the standard set by the faculty. Therefore, by this measure, these questions fell considerably below the standard expected. Further examination of these questions is merited.

The Cronbach alpha for Year 1 RCSI, Year 2 RCSI, QUB and the SHOs was 0.36, 0.29, 0.22 and 0.59 respectively.

Table 1

Top 5 most difficult questions
Difficulty rank of question	1	2	3	4	5
Year 1 RCSI MAC test item	12	3	7	15	1
Year 2 RCSI MAC test item	12	3	7	28	15
SHO MAC test item	3	12	7	6	25

From the results illustrated in Table 1, question item's 3,12 and 7 were the three questions universally difficult across all three groups.

We therefore identified test items 3,12 and 7 as items worthy of further analysis along with test items 1,6,8,10,15,18 and 28. We investigated these questions further with regards the current curriculum to ascertain why they appeared to cause difficulty (see Table 2). We also investigated test items 19,20, and 25 (see Table 3) in more detail as these were paradoxically answered better by students than SHO’s (even though overall, the SHO’s performed much better than the student groups).

Table 2

Analysis of MAC test items identified as ‘particularly difficult’
Item number	Clinical problem questioned	Mentioned in the curriculum?
1	Focal seizure	No
3	Persistent bacterial bronchitis	No
6	Viral induced wheeze	Yes
7	Obstructive sleep apnoea	No
8	Fluid management	Yes
10	Febrile seizure	Yes
12	Viral respiratory infection	No
15	Gestational age	Yes
18	Bronchiolitis	Yes
28	Congenital heart disease	Yes

Table 3

Analysis of MAC test items, which were ‘paradoxically’ answered better by students than SHOs
Item number	Clinical problem questioned	Mentioned in the curriculum?
19	Coarctation of the aorta	Yes
20	New-born screening	Yes
25	Cellulitis	Yes

In Year 1 there were 18 of the 30 items that achieved DI of >0.2. In Year 2, there were also 18 items that achieved DI >0.2. For the SHO group, there were 22 of the 30 items which achieved DI >0.2. With regards item difficulty index (DD), in Year 1, 21 out of 30 items were within the desired range of 0.3-0.7, for Year 2 there were 19/30 and for the BST group there were 14/30 within this range (see appendix 1 for table with full results).

It is unreasonable to expect any undergraduate curriculum to include full coverage of all conditions within paediatrics. Nor would that approach be beneficial to most of the students (only a minority of whom will go on to practice paediatrics). However, it would seem sensible that a student should be expected to acquire the most basic, common, and relevant knowledge first and foremost. The only means by ensuring that they have obtained this knowledge is by assessing it.

When the test item difficulty for the MAC examination was ranked within each group, a Spearman’s rank correlation indicated a positive correlation. This illustrates that, broadly, all three groups found the same questions relatively difficult/easy. This adds an element of reliability to the MAC examination as not just are the overall scores equivalent between the two-year groups, but individual question performance is comparable as well.

It is generally accepted that for an Multiple Choice Question (MCQ) examination, adequate internal consistency is demonstrated by a Cronbach's alpha of at least 0.7 (8). The low level of Cronbach's alpha seen with the MAC examination may be explained by a poor spread of questions. The examination was limited, for the purpose of improving participation, to only 30 questions and thus it is inevitable that there will be aspects of the curriculum that remain untested. It is known that increasing the number of test items will statistically improve the reliability of MCQ's (9). High stakes MCQ examinations would generally have many more than 30 items. When compared with the RCSI official written paper, for example, they use 120 items, which would vastly improve the alpha of the MAC.

Only two-thirds of students passed the MAC exam, whereas 96% of the same students passed their university paediatric examination (2). Their level of paediatric knowledge reflects the university curriculum, therefore there must be significant gaps between the university curriculum and the ‘hidden’ curriculum as determined by non-academic clinicians.

The Royal College of Paediatrics and Child Health (RCPCH) theory examination faculty find that an MCQ item which scores at least 0.2 on the discrimination index is deemed acceptable and therefore a re-usable question. Questions, which score less than 0.2, are not necessarily discarded, rather they go through a review process at one of the regular board meetings where the theory examiners revise the question to improve its future psychometric value. Interestingly, for the original intention of the MAC examination, one in which we hope the candidate proves to have a basic level of knowledge, this measure would not have been as appropriate or useful. The original intention was to design a paper in which the spread of marks would have been less and more bunched around a relatively high gross score, therein, preventing many of the questions displaying good discriminating ability as most candidates would be getting all of them correct. As it turned out, the paper was found to be relatively difficult with a low overall gross mean score and a wide spread of marks for many of the items which show good ability to discriminate candidates.

For a well-prepared group of examinees, item difficulty indices may range from 70-100% (10). Hence, the passing score is usually higher when the examinee group is more able (11). When the difficulty index moves towards high or low from 50%, the discriminating index becomes low. For example, if all the candidates get the question correct or incorrect then there is no way of using that question to discriminate between the best and worst candidates. However, if we omit questions from a potential question bank because the candidates have found it too easy or too hard then, in a sense, the candidates are setting the standard for the papers themselves. We should allow medical professionals to decide what the standard should be (as is the case with our study in using non-faculty clinicians to decide the questions) and then use a rigorous standard setting process such as Angoff to standard set the paper. If it just so happens that the questions are very easy, then this will result in a high passing score and successful candidates must correctly answer a higher-than-normal proportion of questions to pass the exam (as was envisaged for the MAC). Similarly, if the questions happen to be very difficult then this will be corrected for by good standard setting. In that scenario the student need only get a few questions correct to pass.

From the above analysis of the most difficult questions, we can see a pattern emerging. Questions on respiratory paediatrics appear to have been answered poorly. Whilst respiratory conditions are the most common of paediatric presentations, there may be an argument that aspects of the teaching approach at the time did not accurately reflect the real clinical practice for these conditions. The students also seem to regularly have difficulty with questions regarding seizures in childhood despite there being a strong emphasis on seizure disorders and febrile convulsion in the curriculum. The students may not have grasped the subtle but clinically crucial differences between types of seizure presentation. There are multiple examples above where, despite their inclusion in the curriculum, the students appear to have failed to grasp the true clinical relevance of the condition or presentation. Interestingly, for the questions on which the students performed better than the doctors, the questions were quite specific, and the answer can be found directly in the curriculum. This reflects how students learn with limited clinical experience and is a good example of assessment being a key driver of learning.

The concept of a high stake’s assessment with a relatively high passing score breaks from tradition. However, we feel that this model would potentially work very well to ensure that all licensed practitioners have a firm grasp of a finite amount of basic knowledge.

Future research in this area is warranted. Future MAC examinations should have a larger number of test items. This will improve the psychometrics of the examination and will increase coverage of the curriculum. Results from these future studies could help with curriculum development by highlighting specific areas of educational need.

The concept of a MAC examination to test basic medical knowledge is feasible and this is an encouraging first step towards this method of assessment. This method is readily transferable to other medical (and allied health) specialties.

SHO Senior House Officer

MAC Minimum Accepted Competency

RCSI Royal College of Surgeons in Ireland

QUB Queen’s University Belfast

BST Basic Specialist Training

DD Difficulty Index

DI Discrimination Index

MCQ Multiple Choice Question

RCPCH Royal College of Paediatrics and Child Health

Ethics approval and consent to participate

Ethical approval for the project was obtained from the Royal College of Surgeons in Ireland (RCSI) Research Ethics Committee [REC 1129b], the Royal College of Physicians of Ireland (RCPI) Research and Ethics Department [RCPI RESCAF 51] and Queen's University Belfast (QUB) Research and Ethics Committee [18.01]. Written, signed consent was obtained from each of the voluntary participants before they sat the research examination paper.

Consent for publication

Not applicable.

Availability of data and materials

The datasets during and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

No funding was sought or obtained during for the completion of this study.

Author’s contributions

PMcC and NMcC devised the concept of the minimum accepted competency (MAC) examination. PMcC and NMcC developed the methodology of the study. PMcC contacted the clinicians for questions and reformatted the questions to make the MAC examination paper. NMcC provided guidance on how best to analyse the data. AN helped frame the paper within the context of current Irish paediatric education practice. All authors have read and approved the manuscript.

Acknowledgements

We would like to thank each of the paediatric consultants who provided questions to the MAC exam. We would like to thank the RCSI and QUB students, along with the junior paediatric doctors who participated in this study.

Raymond MR, Mee J, King A, Haist SA, Winward ML. What new residents do during their initial months of training. Academic Medicine. 2011;86(10):S59-S62.
McCrossan P, O'Donoghue D, Nicholson A, McCallion N. Comparison of medical student performance in summative undergraduate paediatric examinations and a clinician-designed minimum accepted competency (MAC) assessment. BMC Med Educ. 2021;21(1):197.
Dixon R. Evaluating and improving multiple choice papers: true–false questions in public health medicine. Medical education. 1994;28(5):400–8.
Kaur M, Singla S, Mahajan R. Item analysis of in use multiple choice questions in pharmacology. International Journal of Applied and Basic Medical Research. 2016;6(3):170.
Engelhart MD. A COMPARISON OF SEVERAL ITEM DISCRIMINATION INDICES 1. Journal of Educational Measurement. 1965;2(1):69–76.
Sim S-M, Rasiah RI. Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Annals-Academy of Medicine Singapore. 2006;35(2):67.
Cronbach LJ, Shavelson RJ. My current thoughts on coefficient alpha and successor procedures. Educational and psychological measurement. 2004;64(3):391–418.
Tavakol M, Wilcox RR. Medical education research: the application of robust statistical methods. International journal of medical education. 2013;4:93.
Downing SM. Reliability: on the reproducibility of assessment data. Medical education. 2004;38(9):1006–12.
Thorsteinson TJ. Framing effects on the setting of critical scores for content valid tests. Human Performance. 2006;19(3):201–17.
Livingston SA, Zieky MJ. A comparative study of standard-setting methods. Applied Measurement in Education. 1989;2(2):121–41.

No competing interests reported.

Download PDF

Editorial decision: Major revision
15 Feb, 2022
Reviews received at journal
09 Feb, 2022
Reviewers agreed at journal
18 Jan, 2022
Reviewers invited by journal
08 Oct, 2021
Editor assigned by journal
08 Oct, 2021
Editor invited by journal
07 Oct, 2021
Submission checks completed at journal
07 Oct, 2021
First submitted to journal
05 Oct, 2021

You are reading this latest preprint version

Minimum Accepted Competency Examination: Test Item Analysis

Status:

Version 1

Abstract

Figures

Background

Methodology

Results

Discussion

Conclusion

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Author’s contributions

Acknowledgements

References

Additional Declarations

Supplementary Files

Status:

Version 1