See the published version of this preprint at BMC Medical Education.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research article
Aileen Faherty
National University of Ireland Galway Library
Corresponding Author
ORCiD: https://orcid.org/0000-0002-7016-0099
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background: The reliability of clinical assessments is known to vary considerably with inter-rater reliability a key contributor. Many of the mechanisms that contribute to inter-rater reliability however remain largely unexplained and unclear. While research in other fields suggests personality of raters can impact ratings, studies looking at personality factors in clinical assessments are few. Many schools use the approach of pairing examiners in clinical assessments and asking them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score. Could personality factors have an impact?
Methods: A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality with data collected by questionnaire. A purposeful sample of doctors who examine in the Final Medical examination at our institution was recruited.
Results: Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two-thirds scored high or very high for conscientiousness. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.
Conclusions: The reliability of clinical assessments using paired examiners is comparable to assessments with single examiners. Personality factors, such as extroversion, may influence the magnitude of change in score an individual examiner agrees to when paired up with another examiner. Further studies on personality factors and examiner behaviour are needed to test associations and determine if personality testing has a role in reducing examiner variability.
Keywords
Clinical assessments, Reliability, Examiner Variability, Examiner Factors
Figure 1
Figure 2
Figure 3
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Education.
Version 3
Posted 12 Mar, 2020
No community comments so far
Editorial decision: Accept
On 18 Mar, 2020
Editor assigned
On 05 Mar, 2020
Submission checks complete
On 04 Mar, 2020
Editor invited
On 04 Mar, 2020
No comments provided
Editorial decision: Major revision
On 23 Jan, 2020
Review #2 received
Received 26 Dec, 2019
Reviewer #2 agreed
On 12 Dec, 2019
Reviewer #1 agreed
On 12 Nov, 2019
Review #1 received
Received 12 Nov, 2019
Reviewers invited
Invitations sent on 05 Nov, 2019
Editor assigned
On 29 Oct, 2019
Submission checks complete
On 28 Oct, 2019
Editor invited
On 28 Oct, 2019
No comments provided
Editorial decision: Major revision
On 22 Sep, 2019
Review #3 received
Received 18 Sep, 2019
Reviewer #3 agreed
On 17 Sep, 2019
Review #1 received
Received 16 Sep, 2019
Review #2 received
Received 16 Sep, 2019
Reviewer #2 agreed
On 05 Sep, 2019
Reviewer #1 agreed
On 26 Aug, 2019
Reviewers invited
Invitations sent on 25 Aug, 2019
Editor assigned
On 24 Aug, 2019
Submission checks complete
On 15 Aug, 2019
Editor invited
On 15 Aug, 2019
First submitted
On 10 Aug, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Educational Philosophy and Theory
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
See the published version of this preprint at BMC Medical Education.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Aileen Faherty
National University of Ireland Galway Library
Corresponding Author
ORCiD: https://orcid.org/0000-0002-7016-0099
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Education.
Version 3
Posted 12 Mar, 2020
No community comments so far
Editorial decision: Accept
On 18 Mar, 2020
Editor assigned
On 05 Mar, 2020
Submission checks complete
On 04 Mar, 2020
Editor invited
On 04 Mar, 2020
No comments provided
Editorial decision: Major revision
On 23 Jan, 2020
Review #2 received
Received 26 Dec, 2019
Reviewer #2 agreed
On 12 Dec, 2019
Reviewer #1 agreed
On 12 Nov, 2019
Review #1 received
Received 12 Nov, 2019
Reviewers invited
Invitations sent on 05 Nov, 2019
Editor assigned
On 29 Oct, 2019
Submission checks complete
On 28 Oct, 2019
Editor invited
On 28 Oct, 2019
No comments provided
Editorial decision: Major revision
On 22 Sep, 2019
Review #3 received
Received 18 Sep, 2019
Reviewer #3 agreed
On 17 Sep, 2019
Review #1 received
Received 16 Sep, 2019
Review #2 received
Received 16 Sep, 2019
Reviewer #2 agreed
On 05 Sep, 2019
Reviewer #1 agreed
On 26 Aug, 2019
Reviewers invited
Invitations sent on 25 Aug, 2019
Editor assigned
On 24 Aug, 2019
Submission checks complete
On 15 Aug, 2019
Editor invited
On 15 Aug, 2019
First submitted
On 10 Aug, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Educational Philosophy and Theory
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published article at BMC Medical Education.
Background: The reliability of clinical assessments is known to vary considerably with inter-rater reliability a key contributor. Many of the mechanisms that contribute to inter-rater reliability however remain largely unexplained and unclear. While research in other fields suggests personality of raters can impact ratings, studies looking at personality factors in clinical assessments are few. Many schools use the approach of pairing examiners in clinical assessments and asking them to come to an agreed score. Little is known however, about what occurs when these paired examiners interact to generate a score. Could personality factors have an impact?
Methods: A fully-crossed design was employed with each participant examiner observing and scoring. A quasi-experimental research design used candidate’s observed scores in a mock clinical assessment as the dependent variable. The independent variables were examiner numbers, demographics and personality with data collected by questionnaire. A purposeful sample of doctors who examine in the Final Medical examination at our institution was recruited.
Results: Variability between scores given by examiner pairs (N=6) was less than the variability with individual examiners (N=12). 75% of examiners (N=9) scored below average for neuroticism and 75% also scored high or very high for extroversion. Two-thirds scored high or very high for conscientiousness. The higher an examiner’s personality score for extroversion, the lower the amount of change in his/her score when paired up with a co-examiner; reflecting possibly a more dominant role in the process of reaching a consensus score.
Conclusions: The reliability of clinical assessments using paired examiners is comparable to assessments with single examiners. Personality factors, such as extroversion, may influence the magnitude of change in score an individual examiner agrees to when paired up with another examiner. Further studies on personality factors and examiner behaviour are needed to test associations and determine if personality testing has a role in reducing examiner variability.
Figure 1
Figure 2
Figure 3
Loading...