Setting and participants
The study was done in a real clinical setting, at the Department of Internal Medicine in our university medical centre, in which students are already familiar with video-observation as part of their training. The clerkship internal medicine is the first clinical clerkship in the first master year (M1). The outpatient clinic of internal medicine has two prepared consulting rooms with a fixed camera set-up used for educational purposes.
Participating students were asked to record the history taking during their encounters with real patients, starting in the outpatient clinic from their third encounter. The first encounters could therefore be used to get used to the clinical practice setting. Students were not accompanied by their supervisor during the encounter with the patient, but received the usual feedback from the supervising physician directly after case presentation. Thereafter, the student and supervisor met with the patient together. After the encounter, the supervisor registered a global rating for clinical reasoning as a grade from 1-10 in accordance to usual practice.
For this study, the students would fulfil an extra activity; the completion of a post-encounter form (PEF). This was done after history taking, before receiving feedback of their supervising physician. Students would send their completed PEF digitally to the researchers.
Participating assessors involved in assessment of the students were six principal lecturers, i.e., clinicians with degrees and assignments in medical education. They were not involved as the supervising physicians of the students at the time the recordings were made. These participants received a short instruction about the two rating tools. They were asked to observe the recordings as long as they deemed needed to complete the observation rating tool (ORT). After completing this form, they also completed the post-encounter rating tool (PERT). The time needed to fulfil the assessment was registered by the participants.
Materials
The Observation-rating Tool (ORT) was composed using data from previous qualitative research (6). It consists of eleven pairs of opposite statements about student behaviour related to clinical reasoning. On 5-point scale participants could rate which of the opposite statements was most applicable. (additional file 1)
The student post- encounter form (PEF) and the post-encounter rating tool (PERT) were modified after Durning et al (4). (additional file 2&3) The student post-encounter form consisted of 5 items based on essential parts of clinical reasoning; summary statement, problem list, differential diagnosis, most likely diagnosis and support for the most likely diagnosis. Assessment of the 5 items was done using a 5-point Likert-scale.
Case selection
Twenty students were invited to participate in the study. One case per student was used. The first case that did not meet the exclusion criteria was selected. Cases were excluded when a student was not well visible or when audio quality was low, or the patient was not able to communicate easily, e.g., because of language barrier. Cases that would not induce clinical reasoning, for example when a patient presented with an established diagnosis, were also excluded. After inclusion of 15 cases, selection was stopped. We calculated that 15 cases and 6 observers were needed to reach a power of > 80% to detect an intra-class correlation of 0.30 for the new observation form (7).
Measurements
Feasibility was measured as student completion rate and the time the assessors needed for completing of both the rating instruments and filling out the assessment of satisfaction with the instruments. We included time needed for completion, because it is a limiting factor in clinical practice.
Validity for both instruments was measured as follows: (8) Content validity; For both instruments content validity was measured in two previous studies (4, 6). Internal structure; analysis of internal consistency and generalizability study to explore factors of variance. Response process: This was measured by rater evaluation of both instruments. Relation with other variables: This was done by analysing the association between both instruments.
Reliability of the use of the post-encounter form was already analysed in an OSCE (4), but not for assessment in clinical practice. For both rating tools (PERT and ORT) inter-rater reliability was tested and a generalizability study (G-study) was performed including a decision study (D-study) to assess how many raters would be needed for each instrument to reach an acceptable reliability, since it is known for workplace based assessment tools that reliability, when using one rater, is often low.
Statistics
Cronbach’s alfa was used to calculate internal consistency of scales. Inter-rater agreement was computed using the Intra Class Correlation. We used the ‘two-way mixed model’, because assessments will be done by a selected group of assessors, with measures of ‘consistency’, because the assessment will not be used as a pass or fail test. We used ‘single measures’, since one assessor on the work floor usually performs the assessment during the clerkships.
A Generalizability study was conducted to identify various sources of variance. For evaluation of the observation rating tool a two-facet crossed design with six assessors and 11 items was used. For evaluation of the post-encounter rating tool a two-facet crossed design with six assessors and 5 items was used. A relative G coefficient was computed since we were mostly interested in the rank order of the measurement objects rather than consistency in raw scores.
A Decision study (D study) was performed to forecast changes in G coefficients with alternate levels of facets (assessors and checklist items).
The association between the two rating tools was calculated using Pearson’s correlation coefficient.
We used SPSS version 20 for intra-class correlation and Pearson’s correlation coefficient. Generalizability study was done using the G1 SPSS program (9)
Ethics
Participation was voluntary for all students. All patients were informed about the study, agreed and signed the informed consent form for recording of history taking for educational purposes. Approval of the institutional ethical review board was not required according to the guidelines of the institutional research board based on the Dutch law (WMO)