All Phase 1 students, across two courses (n= 840) were invited to participate in the CLeD-EX field-test. A total of 107 students volunteered to participate and completed the online pre and post-test (response rate: 12.7%). Within that cohort, 52 students were in the intervention group (Colleges A and D) while 55 were in the control group (Colleges B and C). All students were encouraged to submit the completed CLeD-EX forms; 93 completed CLeD-EX forms were received and analysed.
Internal consistency reliability of the CLeD-EX
Cronbach’s alpha for the tutor-evaluation scale was 0.766 and for the self-evaluation scale the Cronbach alpha was 0.451.
Comparison of self-evaluation with tutor evaluation
Item by item paired t-tests (Table 2) were undertaken to compare students’ self-evaluation with tutor evaluation. For all items, except the behaviour “…listens to other’s point of view”, students scored themselves significantly lower than their tutor. Tutors’ evaluations of students’ collaborative behaviours were significantly more positive than students’ self-evaluations for five out of six key behaviours (Table 2). On the CLeD-EX form there was a statistically significant difference between mean summed scores of self-evaluation (Mean=19.4891; SD=1.68049) and mean summed scores tutor evaluation (Mean =21.0746; SD=2.27845); t(92)=1.335; p value ≤0.05, CI= -2.20546, -0.96554. Therefore, there is a significant difference in self and tutor evaluation. Furthermore, Cohen’s effect size value was high (d = 0.79), suggesting that difference has practical significance.
Table 2: Comparison of means between self and tutor evaluation and their level of significance
Behaviour
|
Mean for self-evaluation
(95% Confidence Interval)
|
Mean for tutor evaluation
(95% Confidence Interval)
|
P-value
(two tailed)
|
(Student) well prepared for the learning sessions
|
3.06
(2.96, 3.16)
|
3.52
(3.41, 3.63)
|
.000
|
(Student) willing to work and I am engaged in learning activities
|
3.43
(3.33, 3.53)
|
3.68
(3.58, 3.78)
|
.002
|
(Student) involved in discussion and debate on different ideas
|
3.03
(2.90, 3.15)
|
3.31
(3.16, 3.45)
|
.000
|
(Student) listens to others’ point of view
|
3.74
(3.64, 3.84)
|
3.74
(3.65, 3.83)
|
.892
|
(Student) shares information with group members’ and voice my opinions
|
3.15
(3.03, 3.27)
|
3.38
(3.24, 3.51)
|
.000
|
(Student) reflects on the feedback I receive and respond appropriately
|
3.10
(2.99, 3.21)
|
3.41
(3.29, 3.53)
|
.024
|
Inter-rater agreement: Pearson’s correlation coefficient was calculated to analyse the correlations between the mean scores of student evaluation and mean scores of tutor evaluation. There was a significant correlation between the mean scores of individual items on self-evaluation and mean scores of individual items on tutor evaluation (r=0.887; p=0.019).
In the global, dichotomous item on the self-evaluation scale “overall my skills in collaborative learning are well-developed”, 87 students answered “yes” while six students answered “no.” On the tutor evaluation scale, the global item “Overall this student demonstrates skills which enhance the quality of collaborative learning” was answered positively by 88 out of 89 respondents.
Analysis of utility and feasibility
Two items focussed on utility and feasibility of CLeD-EX were completed by each student after completing the last section (Part 3 Student Reflection) in the CLeD-EX form. For the first item, “I found the CLeD-EX helped improve my skills in collaborative learning”, 81.5% of students agreed. For the second item, “I was provided with useful feedback through the CLeD-EX process”, 92.3% of students responded positively.
The feasibility of the CLeD-EX was ascertained based on the time taken for tutors to give feedback to students. The mean time recorded for completing the evaluation and providing feedback was 6.25 minutes. The minimum time recorded by the tutor on the CLeD-EX to provide feedback was two minutes and the maximum time was 15 minutes. The majority of tutors recorded that the time taken to give feedback to student was five minutes.
Written feedback and student reflections on the CLeD-EX
Analysis of written feedback: Tutors included written feedback in the majority of the CLeD-EX forms (94.6%). Some forms included only comments about positive aspects related to collaboration (12.9%). When analysing the quality of feedback based on the modified framework (38) in the section on the form: ‘areas that can be improved’ the behaviour which needed to be improved was mentioned in 73 forms, while the gap was mentioned in 17 forms and advice on how to develop was included in 32 forms. This finding suggests that the CLeD-EX was successful in facilitating formative feedback by highlighting behaviours that need further development.
A brief review of the content of the written feedback suggested that the most common focus was on the students participating in discussions, for example, “Try to engage more with your peers during small group activities by exploring and discussing issues with them,” and “I suggest you set yourself a goal of contributing one significant idea to the discussion per class.” This was followed by the feedback on student preparation for the small group learning, for example: “Improving preparation can be one approach you could take. This involves reading ahead, which can be useful in providing a sound basis for learning in each SG session.” These findings are consistent with the students’ self-evaluation in which similar behaviours were scored the lowest, also on the tutor-evaluation scale. These findings suggest an important link between the students’ self-evaluation and tutor evaluation. The tutors’ written feedback also emphasises development of these behaviours in the student. Meanwhile, comments for improvement were mainly on aspects related to developing skills to learn more effectively in small group context.
Analysis of Student reflection: In the 93 CLeD-EX forms received, reflections were recorded in 86 forms (92%). This indicates that the structure of the CLeD-EX appears to facilitate students to reflect on the collaborative learning experience. Students in their reflective accounts discussed issues which were being raised through this exercise and their action plans for improvement. In the written reflections relating to the latter, students clearly expressed their desire to improve their behaviour for collaborative learning. This change was observed through comments which indicated an intention to change by developing a new understanding which was gained through this activity.
“I think, in order for me to be able to be more involved in discussions, I need to deepen my knowledge of topics covered in BDGA, beyond lectures and SG (scenario group) activities. In order to do this, I will draw upon textbooks and the extra learning resources in Moodle. This will also allow me to increase my spontaneous engagement in group discussions, as my views will be more informed. I realise now that I should also be more involved in SG debates, and before I was probably too cautious of speaking too much, so I think finding the right balance is key. When an idea is generated by another member, I will be keen to pick it up and discuss it (with the) further” (C-255)
“Primarily, improving my collaborative learning behaviours requires a change in attitude. I need to be less afraid of being wrong when answering questions and remember that as my scenario group is simply a platform for exchanging ideas that I don't need to take personally any disagreements with my own opinions. In terms of practically aiding my capacity to contribute to class, I believe I should come to classes having pre-read the lesson material where possible and conducted background research into topics that are new or unfamiliar to me. This will allow me to pre-identify questions that I can pose to the class. Such preparation will help me to feel more confident in initiating discussion.” (C- 128)
In their reflective comments some students acknowledged the importance of collaborative learning and the usefulness of developing their collaborative learning behaviours. Through this process students have recognised its value for building their skills in collaboration and learning.
“In terms of other aspects of collaborative learning, I will continue to be engaged in all teamwork activities and listen to the opinions of others. In particular, I think our collaborative learning in SGs (scenario groups) will be enhanced if quieter members are drawn out to voice their opinions, so I'll keep this is mind. In short, teamwork is a great way of enhancing the quality of our learning, and not only does it deepen our knowledge, but it develops our broader social interpersonal skills.” (C-255)
Pre-test and post-test evaluation: Impact of CLeD-EX on Students’ Self-Directed Learning (SDL), Critical thinking (CT), and Group Processing (GP)
Comparison of pre-test self-evaluations by students (Table 3) revealed no significant difference between the intervention and control groups.