This preprint is under consideration at BMC Medical Education. A preprint is a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Elizabeth Sinz
Penn State University College of Medicine
Corresponding Author
ORCiD: https://orcid.org/0000-0001-7129-6193
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Introduction: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.
Methods: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.
Results: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.
Conclusions: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.
Keywords
Generalizability, Program Evaluation, Simulation, Assessment, Practicing physicians, Feedback, Competency, Continuing Medical Education
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 2
Posted 08 Jan, 2021
No community comments so far
Editorial decision: Minor revision
On 14 Feb, 2021
Review #2 received
Received 07 Feb, 2021
Reviewer #2 agreed
On 01 Feb, 2021
Review #1 received
Received 08 Jan, 2021
Reviewers invited
Invitations sent on 30 Dec, 2020
Reviewer #1 agreed
On 30 Dec, 2020
Editor assigned
On 17 Dec, 2020
Submission checks complete
On 17 Dec, 2020
Editor invited
On 17 Dec, 2020
No comments provided
Editorial decision: Major revision
On 27 Oct, 2020
Review #2 received
Received 27 Sep, 2020
Review #1 received
Received 24 Aug, 2020
Reviewer #1 agreed
On 27 Jul, 2020
Reviewer #2 agreed
On 27 Jul, 2020
Reviewers invited
Invitations sent on 22 Jul, 2020
First submitted
On 18 Jun, 2020
Editor assigned
On 18 Jun, 2020
Submission checks complete
On 17 Jun, 2020
Editor invited
On 17 Jun, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Learn more about our company.
This preprint is under consideration at BMC Medical Education. A preprint is a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Elizabeth Sinz
Penn State University College of Medicine
Corresponding Author
ORCiD: https://orcid.org/0000-0001-7129-6193
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 2
Posted 08 Jan, 2021
No community comments so far
Editorial decision: Minor revision
On 14 Feb, 2021
Review #2 received
Received 07 Feb, 2021
Reviewer #2 agreed
On 01 Feb, 2021
Review #1 received
Received 08 Jan, 2021
Reviewers invited
Invitations sent on 30 Dec, 2020
Reviewer #1 agreed
On 30 Dec, 2020
Editor assigned
On 17 Dec, 2020
Submission checks complete
On 17 Dec, 2020
Editor invited
On 17 Dec, 2020
No comments provided
Editorial decision: Major revision
On 27 Oct, 2020
Review #2 received
Received 27 Sep, 2020
Review #1 received
Received 24 Aug, 2020
Reviewer #1 agreed
On 27 Jul, 2020
Reviewer #2 agreed
On 27 Jul, 2020
Reviewers invited
Invitations sent on 22 Jul, 2020
First submitted
On 18 Jun, 2020
Editor assigned
On 18 Jun, 2020
Submission checks complete
On 17 Jun, 2020
Editor invited
On 17 Jun, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Introduction: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.
Methods: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.
Results: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.
Conclusions: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.
Loading...