See the published version of this preprint at BMC Medical Education.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research article
Aileen Faherty
National University of Ireland Galway Library
Corresponding Author
ORCiD: https://orcid.org/0000-0002-7016-0099
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background The reliability of clinical assessments is known to vary considerably and inter-examiner variability is a key contributor. This may result in significant differences in scores between comparable candidates, a serious challenge in medical education. An approach frequently adopted to avoid this and improve reliability is to pair examiners and ask them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score.Methods A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality. Demographic and personality data was collected by questionnaire. A purposeful sample of medical doctors who examine in the Final Medical examination at our institution was recruited.Results Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two thirds scored high or very high for conscientiousness. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.Conclusions While the variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12), the reliability statistics for both assessments were comparable. Using paired examiners resulted in a more accurate and robust score than simply averaging two independent examiners scores. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score. These findings could have implications for the organisation and administration of clinical assessments. Further studies with larger numbers of participants might establish if personality testing before choosing examiner pairs could be utilised to help pair examiners and improve examiner variability.
Keywords
Clinical assessments, Reliability, Examiner Variability, Examiner Factors
Figure 1
Figure 2
Figure 3
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Education.
No community comments so far
Editorial decision: Accept
On 18 Mar, 2020
Editor assigned
On 05 Mar, 2020
Submission checks complete
On 04 Mar, 2020
Editor invited
On 04 Mar, 2020
No comments provided
Editorial decision: Major revision
On 23 Jan, 2020
Review #2 received
Received 26 Dec, 2019
Reviewer #2 agreed
On 12 Dec, 2019
Reviewer #1 agreed
On 12 Nov, 2019
Review #1 received
Received 12 Nov, 2019
Reviewers invited
Invitations sent on 05 Nov, 2019
Editor assigned
On 29 Oct, 2019
Submission checks complete
On 28 Oct, 2019
Editor invited
On 28 Oct, 2019
Version 1
Posted 20 Aug, 2019
No comments provided
Editorial decision: Major revision
On 22 Sep, 2019
Review #3 received
Received 18 Sep, 2019
Reviewer #3 agreed
On 17 Sep, 2019
Review #1 received
Received 16 Sep, 2019
Review #2 received
Received 16 Sep, 2019
Reviewer #2 agreed
On 05 Sep, 2019
Reviewer #1 agreed
On 26 Aug, 2019
Reviewers invited
Invitations sent on 25 Aug, 2019
Editor assigned
On 24 Aug, 2019
Submission checks complete
On 15 Aug, 2019
Editor invited
On 15 Aug, 2019
First submitted
On 10 Aug, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
See the published version of this preprint at BMC Medical Education.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Aileen Faherty
National University of Ireland Galway Library
Corresponding Author
ORCiD: https://orcid.org/0000-0002-7016-0099
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Education.
No community comments so far
Editorial decision: Accept
On 18 Mar, 2020
Editor assigned
On 05 Mar, 2020
Submission checks complete
On 04 Mar, 2020
Editor invited
On 04 Mar, 2020
No comments provided
Editorial decision: Major revision
On 23 Jan, 2020
Review #2 received
Received 26 Dec, 2019
Reviewer #2 agreed
On 12 Dec, 2019
Reviewer #1 agreed
On 12 Nov, 2019
Review #1 received
Received 12 Nov, 2019
Reviewers invited
Invitations sent on 05 Nov, 2019
Editor assigned
On 29 Oct, 2019
Submission checks complete
On 28 Oct, 2019
Editor invited
On 28 Oct, 2019
Version 1
Posted 20 Aug, 2019
No comments provided
Editorial decision: Major revision
On 22 Sep, 2019
Review #3 received
Received 18 Sep, 2019
Reviewer #3 agreed
On 17 Sep, 2019
Review #1 received
Received 16 Sep, 2019
Review #2 received
Received 16 Sep, 2019
Reviewer #2 agreed
On 05 Sep, 2019
Reviewer #1 agreed
On 26 Aug, 2019
Reviewers invited
Invitations sent on 25 Aug, 2019
Editor assigned
On 24 Aug, 2019
Submission checks complete
On 15 Aug, 2019
Editor invited
On 15 Aug, 2019
First submitted
On 10 Aug, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published article at BMC Medical Education.
Background The reliability of clinical assessments is known to vary considerably and inter-examiner variability is a key contributor. This may result in significant differences in scores between comparable candidates, a serious challenge in medical education. An approach frequently adopted to avoid this and improve reliability is to pair examiners and ask them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score.Methods A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality. Demographic and personality data was collected by questionnaire. A purposeful sample of medical doctors who examine in the Final Medical examination at our institution was recruited.Results Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two thirds scored high or very high for conscientiousness. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.Conclusions While the variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12), the reliability statistics for both assessments were comparable. Using paired examiners resulted in a more accurate and robust score than simply averaging two independent examiners scores. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score. These findings could have implications for the organisation and administration of clinical assessments. Further studies with larger numbers of participants might establish if personality testing before choosing examiner pairs could be utilised to help pair examiners and improve examiner variability.
Figure 1
Figure 2
Figure 3
Loading...