Video-based Peer Assessment of Collaborative Teamwork in a large-scale interprofessional learning activity

doi:10.21203/rs.3.rs-3340692/v1

Download PDF

Research Article

Video-based Peer Assessment of Collaborative Teamwork in a large-scale interprofessional learning activity

https://doi.org/10.21203/rs.3.rs-3340692/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

The assessment of team performance in the context of large-scale Interprofessional Learning (IPL) initiatives is an important but underexplored area. It is essential for demonstrating the effectiveness of collaborative learning outcomes in preparing students for professional practice. Using Kane’s validity framework, we investigated whether peer assessment of student-produced videos depicting collaborative teamwork in an IPL activity was sufficiently valid for decision-making about team performance, and where the sources of error might lie to optimize future iterations of the assessment.

Methods

A large cohort of health professional students (n = 1218) of 8 differing professions was divided into teams containing 5–6 students. Each team collaborated on producing a short video to evidence their management of one of 12 complex patient cases. Students from two other teams, who had worked on the same case, individually rated each video using a previously developed assessment scale. A generalisability study quantified sources of error that impacted the reliability of peer assessment of collaborative teamwork. A decision study modeled the impact of differing numbers of raters. A modified Angoff determined the pass/fail mark.

Results

Peer assessment of collaborative teamwork was reliable (G = 0.71) based on scoring by students from two teams (n = 10–12) for each video. The main sources of variation were the stringency and subjectivity of fellow student assessors. Whilst professions marked with differing stringency, and individual student assessors had different views of the quality of a particular video, none of that individual assessor variance was attributable to the assessors’ profession. Teams performed similarly across the 12 cases overall, and no particular professions marked differently on any particular case.

Conclusion

A peer assessment of a student-produced video depicting interprofessional collaborative teamwork around the management of complex patient cases can be valid for decision-making about student team performance. Further refining marking rubrics and student assessor training could potentially modify assessor subjectivity. The impact of professions on assessing individual peers and the case-specificity of team performances in IPL settings need further exploration. This innovative approach to assessment offers a promising avenue for enhancing the measurement of collaborative learning outcomes in large-scale Interprofessional learning initiatives.

Peer assessment

collaborative teamwork

Interprofessional learning

student-generated videos

video-based assessment

Generalisability Theory

Teamwork

Collaboration

Professions

Health professional students

Interprofessional Learning (IPL) has evolved into an important component of health professional education. It aims to cultivate collaborative competencies across diverse professions and enhance patient-centred healthcare delivery. [1-4] The value of student teamwork is recognised by IPL competency frameworks, which specifically require opportunities for students to work in interprofessional groups, with an emphasis on collaborative practice and collective decision-making. [5] [6] [7] However, amidst the drive for collaborative excellence, there is a significant and persisting challenge in how to effectively assess the performance of interprofessional teams in authentic large-scale IPL activities.[8] This situation presents challenges, as it hinders universities from delivering high-stakes assessments that adequately align with the importance of achieving collaborative learning outcomes critical for healthcare. [9]

There is a critical gap in the literature; a scarcity of robust assessment methodologies that can comprehensively assess the teamwork, communication, and decision-making skills of students engaged in large scale IPL settings ( N>300). [10] [2] [7] [11] [12] Many current IPL assessments have been designed to test if students can function expertly as competent individuals, or independent practitioners, [13, 14] [15] and focus on short-term knowledge acquisition, impact on attitudes to other professions, and student satisfaction. [16] Several studies report self-assessment, which are not optimal given an individual’s inability to assess such gains accurately. [17] [18] There is a small but growing literature providing validity evidence for the generation and interpretation of team scores intended for a large scale assessment of teamwork performances [18, 19] [20, 21] [15]

In this paper, we extend our research on large scale interprofessional assessment [10] [22] by investigating the psychometrics of a novel IPL assessment that has the potential to be optimised to provide a high stakes assessment of collaborative teamwork. An opportunity to study this arose in a secondary analysis of data from a purposively designed interprofessional activity based around a video-based assessment for learning task. Within our context, multiple student teams embarked on two linked missions: first, to creatively document their collective problem-solving and decision-making processes for a complex patient case, and secondly, to rate the quality of their peers' work as depicted in their videos.

Assessment in IPL

We now summarise the literature on the key issues that informed our choice of a peer rated video-based teamwork assessment task. Following on, we provide an overview of our chosen validity framework and in the methods, describe the measurement model through which we address our research questions. Assessment profoundly shapes student learning and experience in higher education. Assessing team performance encompassing aspects like leadership, communication, collaboration, and decision-making can be complex. Factors that impact any team-based assessment include the specific task (e.g. case management), team composition, and cultural and organizational contexts. [4] Several tools have been developed to observe and measure student team performance either in real-time (in clinical settings or simulation) or video recordings. [7] [19] [15] In the work-based assessment literature, there are similar distinctions to be made between an assessment designed to measure the collective competence of a team or the individual competency of team members. [23] [24] [25] Challenges also arise in assessing student interprofessional team performance due to resource demands and limited curricular opportunities for team-based clinical activities. [2] [8] [26].

Interprofessional collaborative practice operates under the assumption that a student's professional background should not hinder their ability to evaluate peers from other professions. However, established professional perspectives can hinder collaborative learning, [27] [28] [29] particularly in the area of assessment. [2] [7] [14] Currently, there is little evidence as to what extent a student’s profession impacts their assessment of teamwork performance. There is some evidence from the work-based assessment literature that professional designation does influence individual judgement in terms of both stringency and perspective. [30] However, it is not known whether those differences arise during training or employment, or at the student level.

Peer Assessment

Although the affordances of peer assessment in IPL has been recognised for some time, [7] [31] [32] [33] there are several challenges to overcome in implementation. Peer assessment is a recognised teaching and learning strategy to develop student skills in giving and receiving feedback. It can facilitate students to become better self-regulated learners because when they review the work of their peers, they tend to reflect back on their own work and consider ways of improving it. [34] Whilst tutors have only limited time to observe each student, students have many opportunities to observe each other. [35, 36] Students are more likely to see authentic aspects of interprofessional collaboration than tutors when engaged in, for example, practice-based learning in the clinical setting. [37] Several studies have measured uniprofessional small group learning effectiveness based on student self-assessment of particular behaviours. [38] [39] [40] [41] Despite concerns that the quality of evidence on peer assessment assessments is impacted by the varying methodological approaches, [42] it is thought that peer ratings are reliable and perhaps more valid than self- or tutor-based assessments. [33] [43] [44] Both the giving and receiving of peer feedback have good educational impact for the student assessor. [41]

Student-produced video in assessment for learning

There is a long tradition of using video to assess student clinical skills in the laboratory setting [45] [46] and of using video to develop students’ knowledge or skills acquisition. [47] There is a developing literature on student-produced video as a format in the assessment for learning and of learning that calls for educators to embrace student videography. It is a form of assessment that provides students with important graduate capabilities, to actively apply knowledge in authentic contexts, and promote self-reflection and the giving and receiving feedback from others. [48] Examples include science education, [49, 50] health promotion education, [51] [52] [53] and communication skills training in both dentistry. [54] and in pharmacy. [55], and interprofessional settings [56] [57]

In the broader educational literature, studies of student-produced video that include details of the assessment, focus on the adequacy of topic content, communication of the key message, and the technical quality of the video product. [10] A recent literature review of the utility of student-generated video in the health professions suggested projects suffered from having unclear purposes and weak pedagogical considerations which risks undermining the intended learning.[58]

Kane’s Validity Framework

Several interprofessional competency statements have been published internationally that can inform academic content of an IPL assessment rubric. [59-62] To evaluate the validity of the assessment in this study, we used Kane's argument-based validity framework, which emphasizes key inferences as the assessment progresses from a single observation to a final decision. We were interested in four aspects of the validity of the collaborative teamwork assessment tool. [63] [64] These were;

Scoring - the observation of the video was translated into one or more scores,
Generalisation - the observed scores were used to generate an overall test score representing collaborative teamwork in the IPL setting,
Extrapolation - an inference was drawn regarding what the test score might imply for the collaborative teamwork of students, and
Implications - if the scores were credible and reasonably free from error, they could be used as an assessment of collaborative teamwork.

As part of the generalisation argument, reliability refers to the reproducibility or consistency of scores from one assessment to another, and is a necessary, but not sufficient, component of validity. It can be measured by a generalizability co-efficient which estimates multiple sources of error and provides a method which is generalizable to virtually any setting in which the reliability of the instrument could be assessed.[65] As part of the generalisation argument, valid “for decision making” entails generating a defensible standard.

In this paper, we posed two research questions. First, what factors impact the generalisability of a peer assessment of a student-produced video depicting interprofessional collaborative teamwork in a large-scale IPL activity? Second, to what extent does the student assessor’s profession and the patient case on which the team worked impact student ratings?

The Research Context

While the educational design specifics of the IPL activity are discussed elsewhere [59] [66], we offer a concise overview here. In 2015, the implementation of a large-scale IPL event; “The Healthcare Collaboration Challenge,” involved four health faculties and required the mandatory attendance of a number of cohorts of health professional students. At the time of the data collection, there was no systematic preparation of health professional students around IPL, though there were variations in the amount of clinical exposure and thus informal work-based exposure to IPL. Thus, we considered that students were naïve to formal IPL prior to this event.

Students (n = 1220) from eight different health professions were randomly assigned to one of 208 teams in groups of 5–6 students. Most groups consisted of four or more professions. The larger size of the student cohorts of medicine, nursing, and pharmacy meant most groups had two representatives from these professions. Students groups were randomly assigned to one of 12 case-based learning activities. [67] These had been developed by small multiprofessional groups of clinicians from authentic patient cases and were presented in a structured format to a consistent level of difficulty. The cases involved complex continued patient care based in hospital or in the community, requiring input from multiple health professions. Typical cases included a man living with multiple complications of diabetes and cardiovascular disease, and a number of palliative care scenarios. (An example of a case is given in Appendix 1) Prior to the learning activity, students were directed to the University Learning Management system to obtain an overview of the event with information about team allocation, instructions about participation, their allocated case, and guidelines about the assessment tasks. They were informed that they needed to produce a 5-minute video that depicted interprofessional collaborative teamwork across patient issues, team decision-making negotiations, and their management plan for the case study. Student teams were asked to use their own technology (e.g., smartphones and tablets) to film and edit the video. Exemplars from the previous year were available for students to view. A written one-page management plan formed part of the overall assessment but is not considered in this paper. Each of the eight health professions involved had negotiated differing assessment strategies for their participating students, ranging from pass/fail to a proportion of a unit of study core.

Students attended one of three orientation sessions for a 10-minute briefing on IPL's significance and the learning activity requirements. Subsequently, students gathered in teams, introduced themselves, and engaged in icebreaker team-building exercises to foster problem-solving skills (approximately 20 minutes). Following this, students worked on their cases and assessment products (written plans and videos).

Teams were given 48 hours after the face-to-face event to complete the assessment and upload videos to the University’s Learning Management System. Students then peer-assessed two other teams' videos online on the same case, using the rubric for peer assessment.

Rubric development

The need for the video assessment scale in IPL, it’s development and content validation and piloting has been described in detail elsewhere.[10] Collaboration in teams is a complex, voluntary and dynamic process involving several skills” including an appreciation of the patients perspective. [4] In summary, we developed the prototype collaborative teamwork rubric for this purpose by synthesising the contemporary literature on teamwork in IPL, peer assessment, and assessment of student-generated videos. The previous content validation study identified the major domains that were important to staff and students in assessing student-produced videos portraying interprofessional collaborative practice within a large-scale learning activity. [10] The finalized scale was intended to assess four domains;

Patient issues: respect for the patient and family’s experiences and ability to view the situation through the lens of the patient and family,
Interprofessional negotiation: the ability to negotiate with other health professionals in problem solving for the patient.
The management plan: the practicalities of interprofessional care including the coordination required for a well-executed interprofessional management plan.
The effective use of the video medium to engage and communicate the key messages of interprofessional working.

Each domain was rated on a Likert scale of 1–4, with 1 indicating ‘poor:’ and 4, ‘excellent.’ A global rating on a scale of 1–5 of the overall impression of the video in the context of the IPL activity was included, with 5 being excellent. (See Fig. 1) A modified Angoff standard setting procedure for the whole activity was undertaken for the combined weighted score of 60% video mark plus 40% abstract mark giving a pass mark of 49%.

Insert Fig. 1about here.

Data Analysis

Data were cleaned to remove erroneous or missing entries, leaving data from 1,218 of the participating 1220 students.

In Generalisability theory, [68] the G–study provides a multifactorial perspective of the peer rated collaborative teamwork scores by quantifying sources of measurement error that arise from potentially controllable factors. The characteristic of interest in this study was the quality of collaboration displayed in the team video. We were interested to derive the sources of error, which did not relate to this construct of interest. This included assessor stringency/leniency, which is a first-order effect and is defined as the consistent tendency of student assessors to use either the top or the bottom end of the rating scale. Assessor subjectivity is another source of error and refers to variable student assessor preferences in relation to the collaborative teamwork as displayed in the video, and includes how different assessors favour different examples of videos differently over and above their baseline stringency/leniency. [69] We were also interested to know if the students’ profession contributed to error and if the case that students rated, impacted their assessment - either as individuals or by their profession.

We used the General Linear Model within SPSS (version 24) to estimate the influence of all these factors (facets in G-theory) - both first-order effects (e.g., profession and case) and second-order effects (e.g., the interaction between profession and case). A reverse stepwise regression analysis [70] started with the most comprehensive possible model and then excluded redundant variables, and variables contributing less than 3% of the overall variance, one at a time, to derive the simplest fully-explanatory regression model.

Having determined the best fit model for our data in this way, variance estimates were combined [69] to provide an index of reliability (the G coefficient). This allows future iterations of the assessment program to be modified in order to address the main sources of error identified in the initial study. The G-study was based on a partially crossed design with students as assessors (student assessor) partially crossed with the rated student teams (team). Once redundant factors were removed, the reliability of student assessors’ ratings of collaborative teamwork in the fully nested situation (worst case scenario) was calculated in a D-study as:

G = Var_{student assessor}/(Var_team+(Var_{student assessor}/n)+(Var_{student assessor*team}/n)) – where n is the number of student assessors rating a team video.

This assumes that there was a mix of similar professions in each of the teams, but minimal crossing of the student assessors. The actual situation was slightly more favourable since each student assessed the videos of two other teams – providing some crossover. This would yield slightly more favourable reliability but is difficult to model.

Ethics

All research method were carried out in accordance with relevant guidelines and regulations. The University of Sydney Human Research Ethics Committee approved the research. (Protocol number: 2015/320). Informed consent of the students to participate in the study was obtained by the following method. All students were provided with information about the study and their rights as participants. They were given the opportunity to indicate their consent for their data to be used in evaluative research by clicking on an ethical statement within the Learning Management System. .

Data on peer assessment rating of collaborative teamwork were available for 1218 students in 208 teams, using a marking rubric with four checklist items and a global rating, to rate performances on one of 12 patient cases, by teams made up of 5-6 members drawn from 8 health professions. (See Table 1) Medical and nursing students provided scores very close to the mean (3.18 and 3.19 respectively), but diagnostic radiography students had a higher mean score of 3.56 and physiotherapy students, a lower mean score (2.97). No teams failed the expected standard.

Insert Table 1 about here.

During the reverse stepwise regression process, several facets were excluded as redundant. Case (Var_case) and its interactions with item and profession (Var_case*item & Var_{case*profession}) did not make a significant independent contribution to score variance. This means that teams were rated similarly across the 12 cases overall, no particular professional group marked differently on any particular case, and no performance element was easier or more difficult on one case than the others.

The rating item (Var_item) contributed minor score variance (3% of overall variance), and rating item interactions with student assessor, team, and profession (Var_{item*student assessor}, Var_item*team& Var_{item*profession}) made only very small contributions to score variance (≤ 3% of all variance). This means that some performance elements were rated similarly across the 5 items overall and that individuals’ rater stringency/leniency, the stringency/leniency of the various professional groups, and the performance of particular teams did not vary substantially across items.

Finally, whilst professions marked with differing stringency/leniency, (Var_profession = 10%) and individual student assessors had different views of the quality of a particular video (Var_{student assessor*team}= 32%), none of that individual assessor variance was attributable to the rater’s profession (Var_{team*profession}=0%).

The final model included only the variance components shown in Table 2 – all the facets which explain or cause significant independent score variance. The results of the G study with our best-fit model, show as expected, both wanted and unwanted facets contributing to the variation in collaborative teamwork score (Var_team). The largest contributor to error variance was student assessor stringency/leniency (Var_{student assessor}), followed by assessor subjectivity (Var_{team*student assessor}), and profession differences (Var_profession).

Insert Table 2 about here.

In the fully nested D study, we represent the worst-case scenario in terms of sources of unwanted variance that contribute to the reliability of the assessment. The dependability estimates combining the variance components according to the formulae above are given in Table 3. Peer assessment of collaborative teamwork was reliable (G = 0.71), where each team video is rated by twelve student assessors. In our IPL activity, this represents two other teams. The use of three teams to rate one video would have given higher reliability but was not feasible in our setting. ;

Insert Table 3 about here

Summary of Key findings

Our findings estimated the reliability of a peer assessment of a health professional student-produced video of interprofessional collaborative teamwork. Peer assessment in this context was sufficiently reliable, with modest assessor sampling, for decision-making about the quality of the student team performance as depicted by the video. Most of the error variance was attributable to student assessor stringency/leniency and subjectivity. Therefore, strategies to reduce unwanted variance should focus on both elements of student assessor judgement.

We also explored the influence of students’ professions on assessors' judgements about collaborative teamwork during an IPL activity. Some professions rated more stringently than others across the board However, the professional designations were similar in which videos they thought were better and which were worse (same preferences, but differing standards).

In summary, we investigated the validity of the assessment by examining the key claims, assumptions, and inferences that link IPL assessment scores with their intended interpretations using Kane's validity framework. The study provides psychometric evidence to support the use of peer-observed scores to create an overall test score that accurately represents the depiction of collaborative teamwork. This aligns with the Generalization inference in Kane's validity framework. The study also highlights the importance of considering the “Extrapolation,” argument which involves drawing inferences about the impact of professional designations on assessors' judgments, and Implications, which involves applying the scores to inform a decision about pass/fail. Over all this suggests that the scores from the assessment tool appear credible and, after student assessor training, could be used as an authentic assessment of collaborative teamwork. [64]

Comparison with existing literature

Our findings add to the literature in the area of contested and interrelated issues around assessment for the learning of interprofessional teamwork for whole student cohorts. [2] In previous work, we have detailed the educational design of this interprofessional learning activity that engaged all students in authentic team working, whilst at the same time enabling reflection, learning, and demonstrating the creation of new collective knowledge. [71] This study extends the validity evidence of the prior content validation of the video scale [10] [22] by providing psychometric evidence on aspects of a video-based assessment of interprofessional collaborative teamwork. Our findings support the argument that peer assessment might overcome some of the challenges of large scale applications of IPL [64]. Our findings also extend the empirical findings that support the use of peer assessment in uni-professional settings. [72] [73] Not only does peer assessment offer feasibility and learning benefits, but it can also provide reliable scores. Given the purpose of the collaborative video assessment task was to determine whether a team of students have demonstrated the expected standards of interprofessional collaboration, our data support the use of peer assessment for this purpose. It extends research demonstrating the utility of the case based approach for IPL activities for large numbers of students. [74]

Our work provides further insight into the sources of measurement error in using Interprofessional assessment tools [75] and addressing student assessors’ subjectivity and stringency/leniency. Students’ reflection on the purpose and likely outcomes of peer assessment may influence the way in which they make their judgements. They will need to understand how reflection and goal setting can influence their own professional behaviours [76–78] in IPL, as well as their peer rating behaviour of interprofessional collaboration in others. This would be a valuable intervention, especially in relation to graduate attributes.

The impact of profession on rating team performance is an interesting finding. In post qualification settings where ratings involve doctors and nurses around interprofessional collaboration, it appears that professional designation is a significant factor. [30] Our findings that medical and nursing students appear to have similar preferences and standards for team performance suggest that the change may happen post qualification in the workplace. However, we must acknowledge that this observation might be because team performance, rather than individual performance, is being assessed. Further, the IPL activity included first-year medical students and more senior medical students may have different perceptions of IPL team performance.

That teams performed similarly across the 12 patient cases overall, and no particular profession marked differently on any particular case is an interesting finding. There has been a call from some quarters that there is value for all students in team-based learning to undertake the same case, often for pragmatic reasons. [79] Our finding that providing multiple diverse cases albeit of similar complexity, does not negatively impact learning, is reassuring. This is important because there are many educational benefits of a large-scale IPL event where students engage with a range of authentic clinical scenarios. [71] [67]

We have demonstrated reasonable characteristics for an assessment process that engaged large numbers of students with minimal imposition on faculty. We agree with Boud et al., [9] that explicitly focusing on improving assessment practice in IPL may have a large impact on the quality of the collaborative learning agenda. It seems possible that a focus on assessment in IPL reframes the ‘problem’ of assessing large cohorts’ in IPL [2] to one of opportunity and provides a possible way forward in progressing the IPL agenda.

Methodological Strengths and Challenges

This exploratory study is one of the first to identify the factors that influence student assessor judgement in a peer assessment of collaborative teamwork within a large-scale interprofessional learning activity. We acknowledge the potential impact of three important issues in this research on the generalisability of our findings to other settings: the generalisability design, the robustness of the peer assessment tool, and the IPL research context for student learning. Conceptually, G theory treats data from a multivariate random effects perspective. [80] The generalisability study makes use of naturalistic data to fully explore all possible effects or facets and their interactions. However, this is a different paradigm from hypothesis testing which typically fixes and isolates variables for effect testing. We consider that the exploratory paradigm was more appropriate to our research questions. [80]. We acknowledge that for performance assessments, rater error is often outweighed by task sampling error. [81] One way to average out subjectivity in future iterations of the assessment would be to increase the number of tasks (team producing more than 1 video) – this would average out “subjectivity.” However, this was not feasible on our setting. No existing previously validated peer assessment tool of a video of collaborative teamwork was available at the time, therefore we developed a tool to best suit our context. We acknowledge from a construct validity perspective that our data only provide preliminary evidence for a robust validity argument ( scoring, generalisation, extrapolation and implications ) for the video based assessment tool.

We don’t have the data to understand what drives student assessor error in these ratings but plan to investigate these questions using qualitative data. We acknowledge that this is a measure of a collective competency [23] in collaborative teamwork. It does not separate out the individual contribution to collaborative learning. In considering the generalisability to other IPL settings, we have sampled widely across students, differing types of cases, and differing health professions..

Implications for educators

Educators can draw valuable insights from this study to enhance their Interprofessional Learning (IPL) approaches. Firstly, the study underscores the validity of integrating peer assessment into IPL activities. It demonstrates that peer assessment remains reliable even with a modest number of assessors, making it a valuable tool for assessing student-produced videos on interprofessional collaborative teamwork. Therefore, assessment can be leveraged as a driving force in IPL. By shifting the focus to assessment for learning, educators can provide robust evidence of the quality of interprofessional collaborative learning. This shift in perspective reframes assessment challenges as opportunities, providing impetus to the IPL agenda.

Furthermore, educators should emphasize the preparation of students for peer assessment. By cultivating a deep understanding of peer assessment's purpose and implications, students can better assess their peers. Educators should consider interventions that enhance student assessor judgment. Recognizing that biases in peer assessment primarily stem from student assessor leniency and subjectivity, educators can proactively address these influences by encouraging students to critically reflect on their judgement of an IPL performance in the light of the intended criteria.

Incorporating diverse cases of equal complexity in large-scale IPL events need not compromise learning outcomes. Educators can confidently expand the scope of scenarios, enriching students' exposure to authentic clinical situations.

Implications for further research

First, the transformative role of assessment for learning within the IPL landscape emerges as a focal point of this study. The study underscores assessment's capacity to catalyse learning experiences. The shift in perspective from viewing assessment challenges as obstacles to recognizing them as avenues for advancing IPL prompts researchers to consider innovative ways of integrating assessment for learning methods that resonate with the goals of IPL.

Second, the study underscores the validity of peer assessment integration within the IPL paradigm. This observation encourages researchers to investigate the potential of peer assessment as a dependable assessment mechanism across diverse IPL contexts. Given the role of student assessor leniency and subjectivity in shaping assessment outcomes. researchers could explore the underpinning causes of these biases, with a view to conceptualizing strategies that can temper their impact. In our study, most students had had little formal preparation for IPL activities, which accords with the findings of others. [8] In settings where there is formal preparation for IPL, we anticipate a reduction in student assessor subjectivity and the need for fewer student ratings. Further empirical studies on IPL learning activities that use video-based peer assessment are a rich area for further research

A peer assessment of a student-produced video depicting interprofessional collaborative teamwork around management of complex patient cases can be valid for decision-making about student team performance. Each student profession had the same view of good or poor collaborative teamwork but marked to differing standards. The overall standard of collaborative teamwork was similar across a range of cases and the judgements of different professions did not diverge by case. It’s possible that further refining marking rubrics and assessor training could modify student assessor subjectivity. Similarly, the impact of profession on assessing individual peers and the case-specificity of team performances in IPL settings need further exploration. This innovative approach to assessment offers a promising avenue for enhancing the measurement of collaborative learning outcomes in large-scale Interprofessional Learning initiatives.

Ethics approval and consent to participate.

Consent for publication

Not applicable

Availability of materials

Findings from the datasets supporting the conclusions of this article are included within the article. The datasets generated and analysed during the current study are not publicly available due to confidentiality agreements approved by the Human Research Ethics Committee but are available from the corresponding author on reasonable request.

Competing Interests

The following authors have a conflict to declare. CR is a Senior Editorial Board member and IH is an editorial board member of BMC Medical Education. CJ, GN, CG , TC and FH have no conflicts of interest to declare.

Funding

Funding was provided by an internal Educational Improvement Grant through the Office of the Deputy Vice Chancellor (Education) at the University of Sydney.

Authors’ Contributions

CJ, GN, CG, TC, IH, and CR conceived of developed and implemented video-based assessment for the 2015 IPL activity and developed the research design. CJ and CG led the development of the assessment rubric. JC, CR, and FH undertook data analysis, and CR and JC led the interpretation. CR wrote the first draft of the manuscript, and all contributed to revisions. CR, CJ, GN, CG, TC , FH and IH approved the final manuscript.

Acknowledgements

We acknowledge the significant contribution of the Reverend Jim Crossley to an earlier version of this paper. We acknowledge the contribution of the project manager of the HCC Challenge, Stacey Gentilcore, the University of Sydney e-Learning Team, and the various program coordinators responsible for the relevant units of study.

Bogossian F, Craven D: A review of the requirements for interprofessional education and interprofessional collaboration in accreditation and practice standards for health professionals in Australia. Journal of interprofessional care 2021, 35(5):691-700.
Simmons BS, Wagner SJ, Reeves S: Assessment of Interprofessional Education: Key Issues, Ideas, Challenges, and Opportunities. In: Assessing Competence in Professional Performance across Disciplines and Professions. edn.: Springer; 2016: 237-252.
Thistlethwaite JE: Values-based interprofessional collaborative practice: Working together in health care: Cambridge University Press; 2012.
D'Amour D, Ferrada-Videla M, San Martin Rodriguez L, Beaulieu M-D: The conceptual basis for interprofessional collaboration: core concepts and theoretical frameworks. Journal of Interprofessional Care 2005, 19(sup1):116-131.
Collaborative HPA: Guidance on developing quality interprofessional education for the health professions. In.: Health Professions Accreditors Collaborative Chicago, IL; 2019.
Thistlethwaite JE, Forman D, Matthews LR, Rogers GD, Steketee C, Yassine T: Competencies and frameworks in interprofessional education: a comparative analysis. Academic Medicine 2014, 89(6):869-875.
Rogers GD, Thistlethwaite JE, Anderson ES, Abrandt Dahlgren M, Grymonpre RE, Moran M, Samarasekera DD: International consensus statement on the assessment of interprofessional learning outcomes. Med Teach 2017, 39(4):347-359.
Thistlethwaite J, Dallest K, Moran M, Dunston R, Roberts C, Eley D, Bogossian F, Forman D, Bainbridge L, Drynan D et al: Introducing the individual Teamwork Observation and Feedback Tool (iTOFT): Development and description of a new interprofessional teamwork measure. J Interprof Care 2016, 30(4):526-528.
Boud D, Sadler R, Joughin G, James R, Freeman M, Kift S, Webb G: Assessment 2020: Seven propositions for assessment reform in higher education. Sydney, Australia: Australian Learning and Teaching Council. In.; 2010.
Nisbet G, Jorm C, Roberts C, Gordon C, Chen T: Content validation of an interprofessional learning video peer assessment tool. BMC Med Educ 2017, 17(258).
Skinner K, Robson K, Vien K: Interprofessional education: a unique approach to addressing the challenges of student assessment. Journal of Interprofessional Care 2021, 35(4):564-573.
Smeets HWH, Sluijsmans DM, Moser A, van Merriënboer JJ: Design guidelines for assessing students’ interprofessional competencies in healthcare education: a consensus study. Perspectives on Medical Education 2022, 11(6):316-324.
Roberts SD, Lindsey P, Limon J: Assessing students' and health professionals' competency learning from interprofessional education collaborative workshops. J Interprof Care 2018, 33(1):1-9.
Frost JS, Hammer DP, Nunez LM, Adams JL, Chesluk B, Grus C, Harvison N, McGuinn K, Mortensen L, Nishimoto JH et al: The intersection of professionalism and interprofessional care: development and initial testing of the interprofessional professionalism assessment (IPA). J Interprof Care 2019, 33(1):102-115.
Almoghirah H, Nazar H, Illing J: Assessment tools in pre‐licensure interprofessional education: A systematic review, quality appraisal and narrative synthesis. Med Educ 2021, 55(7):795-807.
Dunston R, Forman D, Rogers G, Thistlethwaite J, Yassine T, Hager J, Manidis M, Rossiter C: Curriculum renewal for interprofessional education in health. In.: Office of Learning and Teaching, Australia; 2014.
Reeves S, Fletcher S, Barr H, Birch I, Boet S, Davies N, McFadyen A, Rivera J, Kitto S: A BEME systematic review of the effects of interprofessional education: BEME Guide No. 39. Med Teach 2016, 38(7):656-668.
Hinyard L, Toomey E, Eliot K, Breitbach A: Student Perceptions of Collaboration Skills in an Interprofessional Context: Development and Initial Validation of the Self-Assessed Collaboration Skills Instrument. Eval Health Prof 2018:163278717752438.
Hayes CA, Carzoli JA, LeRoy Robinson J: Development and implementation of an Interprofessional Team-based Care Rubric to measure student learning in interprofessional education experiences: A pilot study. Journal of Interprofessional Education & Practice 2018, 11:26-31.
Hill AE, Bartle E, Copley JA, Olson R, Dunwoodie R, Barnett T, Zuber A: The VOTIS, part 1: development and pilot trial of a tool to assess students’ interprofessional skill development using video-reflexive ethnography. Journal of Interprofessional Care 2023, 37(2):223-231.
Earnest M, Madigosky WS, Yamashita T, Hanson JL: Validity evidence for using an online peer-assessment tool (CATME) to assess individual contributions to interprofessional student teamwork in a longitudinal team-based learning course. Journal of Interprofessional Care 2022, 36(6):923-931.
Jorm C, Nisbet G, Roberts C, Gordon C, Gentilcore S, Chen T: Using complexity theory to develop a student-directed interprofessional learning activity for 1220 healthcare students. BMC Med Educ 2016, 16:199.
Lingard L: What we see and don’t see when we look at ‘competence’: notes on a god term. Advances in health sciences education 2009, 14(5):625-628.
Sebok-Syer SS, Chahine S, Watling CJ, Goldszmidt M, Cristancho S, Lingard L: Considering the interdependence of clinical performance: implications for assessment and entrustment. Med Educ 2018, 52(9):970-980.
Roberts C, Wilkinson TJ, Norcini J, Patterson F, Hodges BD: The intersection of assessment, selection and professionalism in the service of patient care. Medical teacher 2018:1-6.
Symonds I, Cullen L, Fraser D: Evaluation of a formative interprofessional team objective structured clinical examination (ITOSCE): a method of shared learning in maternity education. Med Teach 2003, 25(1):38-41.
Roberts C, Howe A, Winterburn S, Fox N: Not so easy as it sounds: a qualitative study of a shared learning project between medical and nursing undergraduate students. Medical Teacher 2000, 22(4):386-391.
Pirrie Valerie Wilson R.M. Harden J A: AMEE Guide No. 12: Multiprofessional education: Part 2 - promoting cohesive practice in health care. Medical Teacher 2009, 20(5):409-416.
Hall P: Interprofessional teamwork: professional cultures as barriers. J Interprof Care 2005, 19 Suppl 1(sup1):188-196.
Crossley J, Jolly B: Making sense of work‐based assessment: ask the right questions, in the right way, about the right things, of the right people. Medical education 2012, 46(1):28-37.
Thistlethwaite J: Interprofessional education: a review of context, learning and the research agenda. Med Educ 2012, 46(1):58-70.
Simmons B, Wagner S: Assessment of continuing interprofessional education: lessons learned. J Contin Educ Health Prof 2009, 29(3):168-171.
Chhabria K, Black E, Giordano C, Blue A: Measuring health professions students’ teamwork behavior using peer assessment: Validation of an online tool. Journal of Interprofessional Education & Practice 2019, 16:100271.
Nicol D: Guiding principles for peer review: unlocking learners evaluative skills. In: Advances and Innovations in University Assessment and Feedback. edn. Edited by Kreber C, Anderson C, Entwistle N, McArthur J. Edinburgh: Edinburgh University Press; 2014.
Nofziger A, Naumburg E, Davis B, Mooney C, Epstein R: Impact of Peer Assessment on the Professional Development of Medical Students: A Qualitative Study. Academic Medicine 2010, 85:140-147.
Arnold L, Shue C, Kalishman S, Prislin M, Pohl C, Pohl H, Stern D: Can there be a single system for peer assessment of professionalism among medical students? A multi-institutional study. Academic Medicine 2007, 82(6):578-586.
Anderson ES, Ford J, Kinnair DJ: Interprofessional Education and Practice Guide No. 6: Developing practice-based interprofessional learning using a short placement model. J Interprof Care 2016, 30(4):433-440.
Visschers-Pleijers ASF, Dolmans DJM, Wolfhagen IAP, Vleuten CMV: Student Perspectives on Learning-Oriented Interactions in the Tutorial Group. Advances in Health Sciences Education 2005, 10(1):23-35.
de Grave WS, Dolmans DH, van Der Vleuten CP: Student perceptions about the occurrence of critical incidents in tutorial groups. Med Teach 2001, 23(1):49-54.
Khoiriyah U, Roberts C, Jorm C, Van der Vleuten C: Enhancing students' learning in problem based learning: validation of a self-assessment scale for active learning and critical thinking. BMC Med Educ 2015, 15(1):140.
Roberts C, Jorm C, Gentilcore S, Crossley J: Peer assessment of professional behaviours in problem‐based learning groups. Medical Education 2017.
Kamp RJA, Dolmans DHJM, Van Berkel HJM, Schmidt HG: Can students adequately evaluate the activities of their peers in PBL? Med Teach 2011, 33(2):145-150.
Eva KW: Assessing tutorial-based assessment. Advances in Health Sciences Education 2001, 6(3):243-257.
Sullivan ME, Hitchcock MA, Dunnington GL: Peer and self assessment during problem-based tutorials. Am J Surg 1999, 177(3):266-269.
Cant RP, Cooper SJ: Simulation-based learning in nurse education: systematic review. J Adv Nurs 2010, 66(1):3-15.
Bussard ME: Self-reflection of video-recorded high-fidelity simulations and development of clinical judgment. Journal of Nursing Education 2016, 55(9):522-527.
Kelly M, Lyng C, McGrath M, Cannon G: A multi-method study to determine the effectiveness of, and student attitudes to, online instructional videos for teaching clinical nursing skills. Nurse Educ Today 2009, 29(3):292-300.
Jorm C, Roberts C, Gordon C, Nisbet G, Roper L: Time for university educators to embrace student videography. Cambridge Journal of Education 2019, 49(6):673-693.
Hoban G, Nielsen W, Shepherd A: Student-generated Digital Media in Science Education: Learning, Explaining and Communicating Content: Routledge; 2015.
Nielsen W, Georgiou H, Jones P, Turney A: Digital Explanation as Assessment in University Science. Research in Science Education 2018.
Shuldman M, Tajik M: The role of media/video production in non‐media disciplines: the case of health promotion. Learning, Media and Technology 2010, 35(3):357-362.
Krull H: Video Project Brings New Life to Community Engagement. Journal of Nursing Education 2013, 52(8):480-480.
Warren CM, Dyer A, Blumenstock J, Gupta RS: Leveraging Mobile Technology in a School-Based Participatory Asthma Intervention: Findings From the Student Media-Based Asthma Research Team (SMART) Study. American Journal of Health Education 2016, 47(2):59-70.
Omar H, Khan SA, Toh CG: Structured student-generated videos for first-year students at a Dental School in Malaysia. Journal of Dental Education 2013, 77(5):640-647.
Frenzel JE, Skoy ET, Eukel HN: Using student produced videos to increase knowledge of self-care topics and nonprescription medications. Currents in Pharmacy Teaching and Learning 2013, 5(1):44-48.
Olson RE, Copley JA, Bartle E, Hill AE, Barnett T, Dunwoodie R, Zuber A: The VOTIS, part 2: Using a video-reflexive assessment activity to foster dispositional learning in interprofessional education. Journal of Interprofessional Care 2023, 37(2):232-239.
Lyons KJ, Giordano C, Speakman E, Smith K, Horowitz JA: Jefferson Teamwork Observation Guide (JTOG): An instrument to observe teamwork behaviors. J Allied Health 2016, 45(1):49-53C.
Liu Q, Geertshuis S, Gladman T, Grainger R: Student video production within health professions education: A scoping review. Med Educ Online 2022, 27(1):2040349.
Thistlethwaite J, Moran M, World Health Organization Study Group on Interprofessional E, Collaborative P: Learning outcomes for interprofessional education (IPE): Literature review and synthesis. J Interprof Care 2010, 24(5):503-513.
O'Keefe M: Collaborating across boundaries - A framework for an integrated interprofessiona curriculum. In.: Australian Government Office for Learning and Teaching; 2015.
Orchard C, Bainbridge L, Bassendowski S, Stevenson K, Wagner SJ, Weinberg L, Curran V, Di Loreto L, Sawatsky-Girling B: A national interprofessional competency framework. 2010.
Collaborative IE: Core competencies for interprofessional collaborative practice: 2016 update. Washington, DC: Interprofessional Education Collaborative 2016:1-19.
Kane MT: Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement 2013, 50(1):1-73.
Cook DA, Brydges R, Ginsburg S, Hatala R: A contemporary approach to validity arguments: a practical guide to Kane's framework. Med Educ 2015, 49(6):560-575.
Cook DA, Beckman TJ: Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006, 119(2):166 e167-116.
Nisbet G, Gordon CJ, Jorm C, Chen T: Influencing Student Attitudes Through a Student-Directed Interprofessional Learning Activity: A pilot study. International Journal of Practice-based Learning in Health and Social Care 2016, 4(1):1-15.
Thistlethwaite JE, Davies D, Ekeocha S, Kidd JM, MacDougall C, Matthews P, Purkis J, Clay D: The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide No. 23. Med Teach 2012, 34(6):e421-444.
Cronbach LJ, Glaser GC, Nanda H, Rajaratnam N: The dependability of behavioural measurements: the theory of generalisability for scores and profiles. New York: John Wiley; 1972.
Crossley J, Russell J, Jolly B, Ricketts C, Roberts C, Schuwirth L, Norcini J: 'I'm pickin' up good regressions': the governance of generalisability analyses. Med Educ 2007, 41(10):926-934.
Cookson J, Crossley J, Fagan G, McKendree J, Mohsen A: A final clinical examination using a sequential design to improve cost-effectiveness. Med Educ 2011, 45(7):741-747.
Jorm C, Nisbet G, Roberts C, Gordon C, Gentilcore S, Chen TF: Using complexity theory to develop a student-directed interprofessional learning activity for 1220 healthcare students. BMC Medical Education 2016, 16(1):199.
Roberts C, Jorm C, Genticore S, Crossley J: Peer assessment of professional behaviours in problem-based learning groups. Med Educ 2017.
Yang A, Brown A, Gilmore R, Persky AM: A Practical Review for Implementing Peer Assessments Within Teams. Am J Pharm Educ 2022, 86(7):8795.
Connaughton J, Edgar S, Waldron H, Adams C, Courtney J, Katavatis M, Ales A: Health professional student attitudes towards teamwork, roles and values in interprofessional practice: The influence of an interprofessional activity. Focus on Health Professional Education: A Multi-disciplinary Journal 2019, 20(1):8-18.
Lie DA, Richter-Lagha R, Forest CP, Walsh A, Lohenry K: When less is more: validating a brief scale to rate interprofessional team competencies. Med Educ Online 2017, 22(1):1314751.
Kamp RA, van Berkel HM, Popeijus H, Leppink J, Schmidt H, Dolmans DJM: Midterm peer feedback in problem-based learning groups: the effect on individual contributions and achievement. Advances in Health Sciences Education 2014, 19(1):53-69.
van Zundert M, Sluijsmans D, van Merriënboer J: Effective peer assessment processes: Research findings and future directions. Learning and Instruction 2010, 20(4):270-279.
Sluijsmans DMA, Brand-Gruwel S, van Merriënboer JJG: Peer Assessment Training in Teacher Education: Effects on performance and perceptions. Assessment & Evaluation in Higher Education 2002, 27(5):443-454.
Parmelee D, Michaelsen LK, Cook S, Hudes PD: Team-based learning: A practical guide: AMEE Guide No. 65. Medical teacher 2012, 34(5):e275-e287.
Brennan RL: (Mis) Conception About Generalizability Theory. Educational Measurement: Issues and Practice 2005, 19(1):5-10.
Shavelson RJ, Baxter GP, Gao X: Sampling variability of performance assessments. Journal of Educational Measurement 1993, 30(3):215-232.

Table 1. Students with complete data (n=1218) ranked by cohort size rating, showing profession, year of study, program duration, number of students, number of ratings, mean collaborative teamwork rating and standard deviation.

Student profession	Cohort			Number of students	Number of ratings	Mean rating (SD)
	Year of Study	Program
	Year of Study	Duration	Type

Medicine	1	4	Graduate Entry	317	3085	3.18 (0.76)
Nursing	3	3	Undergraduate	312	3010	3.19 (0.77)
Nursing	2	2	Graduate Entry Masters	312	3010	3.19 (0.77)
Pharmacy	4	4	Undergraduate	235	2295	3.29 (0.74)
Pharmacy	2	2	Graduate Entry Masters	235	2295	3.29 (0.74)
Speech Pathology	3	4	Undergraduate	136	1345	3.08 (0.76)
Speech Pathology	2	2	Graduate Entry Masters	136	1345	3.08 (0.76)
Diagnostic Radiography	2	2	Graduate Entry Masters	74	730	3.56 (0.67)
Occupational Therapy	2	2	Graduate Entry Masters	73	725	3.09 (0.69)
Physiotherapy	3	4	Undergraduate	70	700	2.97 (0.68)
Exercise Physiology	2	2	Graduate Entry Masters	26	260	3.04 (0.73)
Total				1218	12150	3.18 (0.73)

Table 2 Variance components of the peer assessment of a video depicting interprofessional collaborative teamwork around management of one of 12 complex clinical cases in health professional students (n=1218) of 8 differing professions.

Component	Meaning	Variance Estimate	Proportion of total Variance
Var_{student assessor}	Student Assessor stringency/leniency	0.164	43%
Var_team	Team performance as rated by the video	0.056	15%
Var_profession	Profession Stringency/leniency	0.037	10%
Var_{team*student assessor}	Student Assessor subjectivity	0.120	32%

Table 3 D study (fully nested) modeling changes in reliability estimates for groups of 5-6 students, assuming a matching of similar professions but no crossing of student assessors.

No. of Student Assessors	G coefficient (fully nested)
4	0.45
6	0.55
8	0.62
10	0.67
12	0.71
14	0.74
16	0.76
18	0.78
20	0.80

Competing interest reported. The following authors have a conflict to declare. CR is a Senior Editorial Board member and IH is an editorial board member of BMC Medical Education. CJ, GN, CG , TC and FH have no conflicts of interest to declare.

Case7HarryWhite.pdf

Download PDF

Editorial decision: Revision requested
27 Jan, 2024
Reviews received at journal
21 Jan, 2024
Reviewers agreed at journal
03 Jan, 2024
Reviewers agreed at journal
12 Nov, 2023
Reviews received at journal
27 Oct, 2023
Reviewers agreed at journal
17 Oct, 2023
Reviewers invited by journal
08 Oct, 2023
Editor assigned by journal
03 Oct, 2023
Editor invited by journal
29 Sep, 2023
Submission checks completed at journal
29 Sep, 2023
First submitted to journal
09 Sep, 2023

You are reading this latest preprint version

Video-based Peer Assessment of Collaborative Teamwork in a large-scale interprofessional learning activity

Status:

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

Introduction

Methods

The Research Context

Rubric development

Data Analysis

Ethics

Results

Discussion

Summary of Key findings

Comparison with existing literature

Methodological Strengths and Challenges

Implications for educators

Implications for further research

Conclusion

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1