Assessment of the performance of orthopaedic residents in clinical practice

doi:10.21203/rs.2.9682/v1

Download PDF

Research article

Assessment of the performance of orthopaedic residents in clinical practice

https://doi.org/10.21203/rs.2.9682/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Objective

Mini-Clinical Evaluation Exercise (Mini-CEX) is a strategy for performing and documenting direct observation and formative assessment of clinical skills. This assessment method promotes identify deficiencies in history taking, physical examination and counseling skills. Furthermore, Mini-CEX is easy to implement and less cost. Thus, our objective was to evaluate the performance of orthopaedic residents in while conducting clinical examinations.

Methods

Mini-CEX was applied by three faculties in four different moments. The instrument was adapted by the authors for use in orthopaedics, with the development of descriptors for each evaluated skill. Supervisors were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors.

Results

The mean scores obtained in the 4 evaluations for each of the 21 residents reveal improvement in the performances of residents in all skills assessed from the first to the fourth meeting.

Conclusions

We have found that the performance of orthopaedic residents presented a satisfactory evolution, with progressive improvement in all skills.

Educational Philosophy and Theory

Physical examination

learning

teaching

orthopedics

Reliable methods for assessing clinical resident’s performance is a problem faced by teaching specialists worldwide¹. Direct observation is a good tool for analyzing them in the workplace, allowing for the opportunity to observe what they know and can do. However, although faculties and residents work in the same place, observation is traditionally informal and carried out inconsistently in almost all medical specialties².

To encourage that observations be performed by the faculties, the American Board of Internal Medicine proposed the Mini-Clinical Evaluation Exercise (Mini-CEX), a 10- to 20-minute direct-observation assessment of trainees’ clinical skills. The evaluation takes place during trainee-patient meetings; interaction between the resident and a patient is observed, after which the evaluator provides the resident with feedback in the form of one-page written assessment rating such skills as “overall clinical competence” and “humanistic qualities/professionalism” and signed by both the observer and the trainee^3.

This method has been used in such specialties as cardiology⁴, anesthesia⁵, pediatrics⁶, internal medicine⁷ and chiropractic⁸ However, no studies on its use in orthopaedics have been found. Thus, our objective was to evaluate the performance of orthopaedic residents in while conducting clinical examinations.

The study was a longitudinal prospective study, carried out from January to July 2016, in two residency programs in orthopaedics, accredited by the Ministry of Education and recognized by the Brazilian Society of Orthopedics and Traumatology, in a capital of the Central-West region of Brazil.

Participants

Nonprobability sampling was employed. We included the postgraduate first-year (R1), second-year (R2), and third-year (R3) orthopaedic residents in the two residency programs that agreed to participate in the study, and the participants signed informed consent forms. Excluded from the study were those belonging to (i) nonaccredited services of the Brazilian Society of Orthopaedics or (ii) trainees of subspecialties of orthopaedics (R4), such as spine, knee, shoulder, and foot and ankle surgery.

Data Collection

The Mini-CEX was applied by three faculties of the respective hospitals in orthopaedics. They were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors. The Mini-CEX was used during individual resident-patient interactions in the ready care, infirmary, and ambulatory departments. Its application occurred on dates formally scheduled by the appraisers to ensure that all residents went experienced the evaluation on the same day, in their respective places of operation. Each resident was evaluated at four different times, with an interval of at least 30 days between evaluations.

Instrument Used

The Mini-CEX, created by the American Board of Internal Medicine, presents six clinical skills, as follows: (i) medical interviewing (clinical history); (ii) physical examination, (iii) humanistic quality/professionalism; (iv) clinical judgment; (v) counselling skills (communication and advice); and (vi) organization/efficiency. The instrument was adapted by the authors for use in orthopaedics, with the development of descriptors for each evaluated skill. The process of adaptation of the instrument took place in specific meetings, where four faculties with more than 10 years of involvement with the residency program in orthopaedics, and two heads of residence of this specialty were present. After five meetings with an average duration of 120 minutes, the instrument was adapted and defined such that each skill had 9 descriptors. If the objective of the respective descriptor was achieved, the resident received a point, respecting the original’s maximum of nine points. The scoring scale was divided into 1–3 (unsatisfactory); 4–6 (satisfactory), and 7–9 (outstanding). The methodology used by Abadie et al. was used to develop the descriptors.

Outcome

The dependent variable was the performance of the orthopaedic residents in the physical examination, evaluated by the Mini-CEX instrument.

Independent Variables

The following variables were used: age (in years lived); gender (male/female), and year of residency (R1/R2/R3).

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Data Analysis

The data were recorded in a Microsoft Excel^® spreadsheet, and the statistical analyses were performed using the SPSS^® program, version 23. The quantitative variables were presented as mean and standard deviation. The total Cronbach’s alpha coefficient was calculated at each scale to evaluate the internal validity of the Mini-CEX instrument. Normality was verified using the Shapiro-Wilk test. Nonparametric statistics were applied if data were not normal. The evolution of the residents (R1, R2, and R3) throughout the duration of the study was performed using the Friedman test, followed by analysis of multiple post-hoc comparisons. The probability of rejecting the null hypothesis was 5%.

Ethical Considerations

The work was approved by the Ethics and Research Committee of the Hospital das Clínicas of the University of Goiás, under the number 1,326,398, on November 9, 2015, and registered in the Platform Brazil, CAAE protocol number: 49761515.1.0000.5078. All participants included in the study agreed to participate and signed the Informed Consent form.

The total number of residents was 33, of which 12 were excluded, 3 were R4, and 9 were nonaccredited. Thus, the final sample consisted of 21 residents, 7 in each year of residence, and all male. The mean age was 25.86 years (± 2.27) for the R1s, 27.50 years (± 1.74) for the R2s, and 27.14 (± 1.97) for the R3s.

The total Cronbach’s alpha coefficient was 0.88 when all the skills of the Mini-CEX instrument were considered, with a 95% confidence interval ranging from 0.79 to 0.94. The “clinical reasoning” and “clinical history” skills were the ones with the highest values, 0.83 (0.67–0.92 CI) and 0.76 (0.52–0.89 CI), respectively.

The general evaluations using the Mini-CEX of the performance of residents during the clinical examinations on the 4 test days are shown in Table 1. A comparison of the scores from the first evaluations with the last ones showed significant improvement of all the clinical skills.

The mean scores obtained in the 4 evaluations for each of the 21 residents are presented in Table 2, broken out individually by residency year (R1, R2, R3) and the skills. The results reveal improvement in the performances of the R1, R2, and R3 residents in all skills assessed from the first to the fourth meeting.

A comparison of the mean scores of the Mini-CEX scores of the 4 meetings showed that there was a significant improvement in the performance of the residents during the evaluations. It is interesting to note that these data contrast with what exists in the literature. A systematic review of 119 papers on the tools for evaluation and direct observation of clinical skills, using the same instrument, showed that there was no improvement in clinical skills and patient care⁹.

Another study, published in Pediatrics, used the Mini-CEX during 4 to 6 evaluations of 23 residents over a year. In that study, the clinical skill that showed the most improvement was humanistic qualities/professionalism, and the skill with the least improvement was physical examination¹⁰. This partially corroborates our findings as it supports a general progression over the course of the evaluation; however, it differs from our findings in that we found significant improvement in the residents’ physical examination skill: the scores increased from 2.67 for the first test day to 6.57 for the last test day. Physical examination and medical interviewing were the skills that our study found to improve most over the course of the evaluations. This can be explained by differences in feedback styles and focus (in our study, more feedback was given on physical examination) between the different Mini-CEX studies.

Other medical specialties also presented satisfactory results in relation to physical examination skills. Another study’s evaluations of pediatric residents found an average of 6.1 for this skill⁶. Similar results were found in 108 cardiology residents, with a mean of 7.1, 7.5, 7.5, and 8.0 found in R1, R2, R3, and R4 residents, respectively⁴.

The present study shows that the average score for physical examination skills for the R1s in the first test day was the lowest one found when compared with the R2s and R3s. This can be attributed to the R1s’ relative newness to the specialty, less time in theoretical/practical classes, and less observation by supervisors in their consultations. In addition, the first evaluation happened shortly after the R1s’ entrance to the specialty of “orthopaedic”, when they were only equipped with the knowledge acquired at the undergraduate level; this suggests a lack of specialty teaching during medical school and the need to correct this gap during medical residency.

It can be observed that the residents showed improvement in their clinical skills during the serial evaluations; however, the R2s and R3s did not evolve from the third to the fourth assessment in the area of physical examination skills, probably due to having been in the specialty for a longer time and having already achieved greater skills and satisfactory scores in that period of the study.

It is important that orthopaedic residents perform optimally, especially in the physical examination. To achieve this goal it is necessary for faculties to use the best teaching and learning resources possible. Teaching is most effective when residents are involved both physically and mentally—namely, through direct involvement in patient care—by encouraging residents to take notes while studying, especially about relevant questions to ask supervisors and/or patients. Another way to improve residents’ performances would be to focus on their “development zone”—that is, to start from the exact point of their gaps or misconceptions rather than teaching what is already known. Thus, it is crucial to identify the limits of residents’ knowledge¹¹.

Teaching the orthopaedic physical examination is challenging. There are a number of variables involved in the process, such as the clinical skill of the supervisor, the willingness of the patient to be evaluated in a group as an illustrative example as physical examination signs are taught, and the interests of the residents. The best strategy for teaching this skill is to start a supervisor with top-level skills and a strong aptitude for teaching. Attributing this function to someone without either ability would only perpetuate residents’ bad habits.

Another effective strategy to achieve a quantitative increase skill is teaching orthopaedic physical examination at the bedside. This is perhaps one of the best to promote its improvement. This learning scenario occurred routinely in the 1960s, but it became infrequent in the 1990s¹². Physical examination skills cannot be taught effectively in theory lectures without the presence of a real or simulated patient.

This study becomes relevant in the context of a lack of research in this area, showing the positive evolution of the performance of residence with the Mini-CEX, as no similar studies were found related to orthopaedics. Our findings contribute to the knowledge of this subject, with special significance for our target population. We performed a longitudinal and prospective study, without memory bias, which gave the data obtained greater credibility. Another strength of the study would be the chosen evaluation instrument. The Mini-CEX was designed, based on (and is used in) real situations, to be a comprehensive yet streamlined tool for evaluating clinical skills, and is therefore a distinguishing instrument. The Mini-CEX also enables evaluations in different scenarios and with different levels of complexity.

Because there was no previous sample calculation of the subjects to be evaluated, the final nonprobability sample was considered small, coming from only two hospitals. Statistically, nonrandom selection procedures may not guarantee representativeness; consequently, our findings should not be generalized to a broader population. Despite the need for new studies on the validity and feasibility of the Mini-CEX in the Portuguese language, this study did demonstrate that the instrument tested has a high reliability and internal consistency.

This study found that the performance of residents of orthopaedics and traumatology during clinical examinations presented a satisfactory evolution, with progressive improvement in all skills, regardless of the year of residency. The research favors the development of other studies to broaden the understanding of the phenomenon studied or to empirically confirm the results obtained.

Ethics approval and consent to participate

Consent for publication

Not applicable.

Availability of data and materials

The datasets analyzed during the current study is available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This work did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. No funding was obtained for this study.

Authors’ contributions

MA and PR. made substantial contributions to the study design, acquisition of data, analysis and interpretation of data and have been involved in revising the manuscript critically for important intellectual content and has also given final approval of the version to be published. RF. has also been involved in writing the manuscript, revising it critically and giving the final approval of the version to be published. All authors read and approved the final manuscript.

Acknowledgements

We would like to thank all the residents and faculties that participated in this study and accepted to be observed during the evaluation sessions and fill the questionnaire. This study has been possible thanks to their generosity in sharing the beliefs and opinions about this assessment approach.

Pitts D, Rowley DI, Sher JL. Assessment of performance in orthopaedic training. J Bone Joint Surg. 2005;8787(9):1187–91.
Fromme HB, Karani R, Downing SM. Direct observation in medical education: a review of the literature and evidence for validity. Mount Sinai J Med. 2009 Jul 29;76(4) 365–71.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The Mini-CEX: A method for assessing clinical skills. Ann Intern Med. 2003 Mar 18;138:476–81.
De Lima AA, Barrero C, Baratta S, Costa YC, Bortman G, Carabajales J, Conde D, Galli A, Degrange G, Van DER Vleuten C. Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Med Teacher. 2007 Aug;29:785–90.
Weller JM, Jolly B, Misur MP, Merry AF, Jones A, Crossley JGM, Pedersen K, Smith K. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth. 2009 Mar 31;102(5):633–41.
Abadie, Y, Batolla, J, Zubieta, A, Dartiguelongue, J, Pascual C, Cost CE, Vassallo JC, Rodríguez S. Uso de descriptores durante la implemantación de Mini-CEX en la residencia de pediatría. Medicina (Buenos Aires). 2015 Oct;75:289–96.
Malhotra S, Hatala R, Courneya C-A. Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise. Med Teacher. 2008 Jul 3;30:414–9.
Paravicini I, Peterson CK. Introduction, development, and evaluation of the miniclinical evaluation exercise in postgraduate education of chiropractors. J Chiropr Educ. 2015 Mar;7899(0):1–7.
Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009 Sep 23;302(12):1216–326.
Goel A, Singh T. The usefulness of mini clinical evaluation exercise as a learning tool in different pediatric clinical settings. Int J Appl Basic Med Res. 2015 Aug;5:32–4.
Pinney SJ, Mehta S, Pratt DD, Sarwark JF, Campion E, Blakemore L, Black KP. Orthopaedic surgeons as educators. Applying the principles of adult education to teaching orthopaedic residents. J Bone Joint Surg Am. 2007 Jun;89:1385–92.
Ende J. What if Osler were one of us? Inpatient teaching today. J Gen Intern Med. 1997 Jan 28;12(2):41–8.

Table 1 – Assessment of clinical skills using the mini-CEX in four meetings

Competence	Meetings (Mean ± Standard deviation)				p*
Competence	First	Second	Third	Fourth	p*
Physical examination	2,67 ± 1,02d	3,71 ± 0,96c	5,86 ± 0,96b	6,57 ± 1,25a	< 0,001
Medical interview skill	2,86 ± 1,39d	3,95 ± 1,02c	6,00 ± 0,95b	7,00 ± 0,89a	< 0,001
Clinical judgment	4,19 ± 1,33c	4,90 ± 0,83c	6,57 ± 1,16b	7,38 ± 0,86a	< 0,001
Humanistic quality	4,14 ± 1,24c	5,19 ± 1,03b	6,14 ± 1,11a	6,76 ± 1,04a	< 0,001

Friedman test followed by Post-Hoc analysis.

*Different letters indicate significant differences.

Table 2 – Mean scores obtained in four meetings according to resident’s postgraduate year

Competence	Y*	First	Second	Third	Fourfth	p*
Physical examination	R1	1,86 ± 1,07d	3,00 ± 0,58c	4,86 ± 0,90b	6,29 ± 0,49a	<0,001
	R2	3,17 ± 0,75b	3,67 ± 1,21b	6,33 ± 0,52a	6,83 ± 0,75a	0,001
	R3	3,00 ± 0,76b	4,38 ± 0,52b	6,38 ± 0,52a	6,63 ± 1,92a	0,001
Medical interview skill	R1	1,43 ± 1,13c	3,00 ± 0,58c	5,14 ± 1,07b	6,29 ± 0,49a	0,001
	R2	3,50 ± 0,84c,b	4,50 ± 1,22b	6,17 ± 0,41a	7,00 ± 0,89a	<0,001
	R3	3,63 ± 0,92c	4,38 ± 0,52c	6,63 ± 0,52b	7,63 ± 0,74a	0,001
Clinical judgment	R1	2,86 ± 1,07c	4,29 ± 0,49c	5,57 ± 1,13a	6,57 ± 0,53a	0,001
	R2	4,67 ± 1,03c	5,33 ± 0,82c	6,67 ± 0,82b	7,83 ± 0,75a	0,001
	R3	5,00 ± 0,76b	5,13 ± 0,83b	7,38 ± 0,74a	7,75 ± 0,71a	<0,001
Humanistic quality	R1	3,71 ± 1,11b	4,86 ± 1,21b	5,71 ± 1,11a	6,71 ± 1,11a	0,001
	R2	4,17 ± 1,47b	5,00 ± 1,26b	6,00 ± 1,10a	6,33 ± 0,52a	0,002
	R3	4,50 ± 1,20d	5,63 ± 0,52c	6,63 ± 1,06b	7,13 ± 1,25a	<0,001
* Friedman test followed by PostHoc analysis.

*Y – postgraduate year

Different letters indicate significant differences. R1 – first year resident, R2 – second year resident, R3 – third year resident

Download PDF

Version 1

posted

You are reading this latest preprint version

Assessment of the performance of orthopaedic residents in clinical practice

Status:

Version 1

Abstract

Background

Methods

Results

Discussion

Conclusions

Declarations

References

Tables

Status:

Version 1