Pilot Study: Agreement Between Self-, Peer- and Tutor- Assessment in a Pre-Clinical Skills Programme

PURPOSE/OBJECTIVES: As dental education evolves, a shift towards self- and peer-assessments, producing dental professionals who are reective, life-long learners, is evident. This study aims to establish the agreement of competency between and amongst students, and their course tutors, during a pre-clinical skills programme. METHODS: Following cavity preparation of plastic teeth during a pre-clinical skills session, students assessed their own cavity preparations and those of their peers. Two course tutors undertook assessment of the same prepared teeth. Plastic teeth were used during the pre-clinical skills sessions and assessed using a WHO CPITN probe and light. Overall scores of competent (C) or not competent (NC) was assigned to each tooth. The data were collected and analysed using Fischer's exact test. RESULTS: No statistical signicance was found between self-, peer- or tutor-assessments. Tutors’ assessments were in agreement. Peer- and self-assessment agreed with tutor-assessment two-thirds of the time. When not in agreement with tutor-assessment, peer- and self-assessment had a tendency to underestimate scores. CONCLUSION: Whilst no statistical signicance was found, tutor-assessment had a high degree of agreement. Peer- and self-assessment appeared to agree with the tutor-assessment two-thirds of the time. When not in agreement with the tutor, students tended to underestimate themselves and their peers.


Background
Dental/medical education is a eld which is continually evolving. Over the past few decades, the focus has shifted from summative assessment towards methods of formative assessment. This change will meet the needs of the dental/medical profession in the long-term to produce more re ective clinicians and, in turn, lifelong learners. [1][2][3][4] The three main methods of assessment are self-, peer-and tutor-assessment. The ideal of producing lifelong, re ective learners appears simple in theory, though in practice the self-re ection necessary to drive an adult towards the learning needed to improve their skills may not be so straightforward.
Some studies have shown that most professionals are not good at assessing themselves; however, these reviews determined that the ability to self-assess is key in becoming, and remaining, a successful professional in the dental/medical eld. 2,5−10 On the other hand, peer-assessment appears to hold more weight in terms of validity and is usually in line with tutor ratings. 5,6,11−17 Tutor-assessment, both summative and formative, is the conventional method of assessment.
Pre-clinical skills laboratory course component forms an essential part of the Bachelor of Science in Oral Health Science (BSc OHS) degree at most universities within the United Kingdom. Currently, within Edinburgh, tutor-assessment of the pre-clinical session is utilised, although no self-or peer-assessments are included in this component.
The comparison between modes of assessment has been a topic of research over the past few decades.
In the eld of dentistry, speci cally in the area of pre-clinical skills, there is limited research which compares the different assessment outcomes. A study in 2001, found that the methods of self-and peerassessment are under-utilised in the clinical training of dentists in the restorative eld. 18 With the drive for self-directed learning in full swing, it is important not only to impress these methods of assessment onto clinical programmes in dentistry, but also to ensure their proper execution and effective utilisation.
Another study compared the self-, peer-and tutor-assessments in relation to a clinical procedure and found, in line with most research, that self-assessment had poor correlation with tutor scores. Peerassessment scores, however, were consistent with tutor scores. They utilised checklists and global scoring which appears to give more reliable results in most studies. 15 In 2015 a study undertaken in Dundee compared the assessment outcomes of self-, peer-and tutorassessment in a pre-clinical laboratory setting. This study found that students needed more training to enhance both their self-and peer-assessment skills, as there was poor correlation between the three assessment types, with students over-estimating their own, as well as their peers, performance in comparison to the tutor's evaluation. 19 This outcome would suggest that students need more training in assessment, or that they do not hold the level of expertise required to measure and evaluate to a satisfactory level.
This project aims to encourage the incorporation of self-and peer-assessment into the pre-clinical curriculum as an adjunct to the conventional tutor-assessment in the pre-clinical laboratory component of the BSc OHS course.

Methods
Ethical approval was obtained from The University of Edinburgh, CMVM EREC (College of Medicine and Veterinary Medicine Educational Research Ethics Committee).
Students were recruited by distributing information regarding the study a week prior to the data collection sessions, and written consent forms were completed on the day of the data collection. At the planning stage of the study there were ten students eligible to progress to second year and participate in this research project, however, at the time of data collection only six students remained on the course.
In 2017-2018, second-year BSc OHS students performed four cavity preparations/restorations on primary and permanent plastic teeth: a single-surface adhesive occlusal cavity preparation and restoration; a single-surface adhesive labial cavity preparation and restoration; a multi-surface adhesive cavity preparation and restoration, and; a multi-surface amalgam cavity preparation and restoration.
All prepared teeth were placed in a bag labelled with the student's university assigned code, Undergraduate Graduate (UG) followed by a two-digit number and stored.
An example of the code is: UG67010301, this would refer to a tooth prepared by student UG67 (UG67), primary dentition (01), multi-surface adhesive cavity preparation and restoration (03) and it was the rst tooth prepared in that manner (01). The next tooth to be identi ed as a multi-surface adhesive cavity preparation and restoration in the primary dentition by student UG67 would be numbered as: UG67010302.
Data Collection was carried out in two phases: (1) Student phase (March 2018): using standardised descriptors the students assessed their own prepared plastic teeth and those of a randomly allocated student peer, (2) Tutor phase (June 2018): two clinical skills tutors also assessed the same teeth using the same descriptors.
Each student assessed their own prepared plastic teeth as a self-assessment. Using a computer randomisation technique on Excel, the student's university numbers were randomly cross matched to select the student whose teeth they would assess as peers.
The plastic teeth to be assessed were given to each student along with the data collection sheet. They were asked to assess the teeth presented to them according to the criteria on the data collection sheet using a WHO CIPTN probe to aid them in identifying the characteristics of the tooth to be measured (cavity depth/ width, etc) and a light in the clinical skills laboratory.
The second phase of data collection involved two of the BSc OHS tutors assessing the same teeth in the same manner and completing data collection sheets accordingly.
The data was analysed by Fischer's exact test (IBM SPSS Statistics version 22) and the p-value was set at 0.05.

Results
Five students and two tutors took part in this project.
No statistical signi cance was found in the data when comparing self-to tutor-assessment, or peer-to tutor assessment, or self-to peer-assessment (Table 1 -8). Further analysis using the data from the second tutor was not undertaken as the tutor-assessments were in agreement. However, patterns could be seen from the raw data.        Of the sixty-six plastic teeth assessed, the two tutors agreed on their grading for sixty of the prepared teeth.
Self-assessment scoring aligned with the tutor-assessment scoring for forty-three out of sixty-two of the prepared teeth, only slightly lower than the agreement between peer-assessment and tutor-assessment (thirty-nine of the fty-four prepared teeth).
When not in agreement with the tutor scores, peers gave scores lower than that of the tutor in ten of the fty-four prepared teeth and higher than the tutor in ve of the fty-four teeth prepared. Similarly, selfassessment was lower than that of the tutor in thirteen of the sixty-two teeth scored, and higher in six of the sixty-two scores given.
When comparing self-assessment with peer-assessment, the scores were in agreement for the assessment of twenty-nine out of forty-six of the prepared teeth. Peer-assessment scores were lower than self-assessment scores in seven out of forty-six of the evaluated teeth and self-assessment scores were lower than peer-assessment scores in ten out of forty-six of the evaluations.
In summary, it could be said that course tutors were in agreement and that peer-and self-assessment appeared to agree with the tutor-assessment two-thirds of the time. When they were not in agreement with the tutor, students tended to underestimate themselves and their peers.

Discussion
Although no statistical signi cance was seen in this study, this could be due to the small sample size.
In the Edinburgh Dental Institute, the pre-clinical skills courses form part of both the undergraduate and postgraduate programmes. The undergraduate programme was chosen for this project due to its structured format and the number of students being higher than that of individual postgraduate programmes. Although the number of participating students was quite low, the number of prepared plastic teeth which were included in the study per student were increased to try to compensate for this de cit.
In total sixty-six prepared plastic teeth were assessed. The prepared teeth were labelled with a code which included the university student code. Whilst an effort was made to randomise which student assessed which peer, the small group made it likely that the students knew who they were assessing. This, along with the small sample size, were some of the limitations of this study. Further limitations include the inexperience of the students with using the descriptors; with practice and training the results may have been different.
One of the patterns in the data collected which was consistent with the ndings of previous studies was the agreement between tutor-assessment and peer-assessment. 15,19 It has been said that under-performing students are more likely to over-estimate their performance and competent students are able to accurately assess their performance. 7 Therefore, these students who assessed their performance accurately (70% of the time) and when not in agreement with their tutor, under-estimated themselves are likely to be competent in their eld. 9,10,20−24 The level of agreement between peer-and self-assessment was over 60% in this study and this added to their value as a positive indicator of their competency to assess, whether it be themselves or their peers. 11,25 One study mentioned that if there is a lag time between the task taking place and the assessment of the task it can lead to less agreement between assessors. If this study had taken place on the same day as the session in which the plastic teeth were prepared, the agreement in results might have been higher. 6 As students become more con dent in their competence to accurately assess themselves and their peers, it will mean more frequent feedback could be provided and this will enable more dynamic learning as a whole for the students and with less effort on the part of the tutor. 11,21 The value of self-assessment, once recognised, makes the process less of a hassle in the eyes of the students, so reporting the ndings back to students could have a positive impact on their attitude towards self-and peer assessment. 26 There is scope for repetition of the study with the inclusion of similar courses in other universities to increase the number of participating students.

Conclusion
Self-and peer-assessment is an important method of assessment which can give similar results to that of tutor-assessment. These skills of assessment will enable a dental student to progress through their career as a self-re ective dental professional who is able to give valuable feedback to their colleagues.
Students tend to underestimate themselves and their colleagues, while tutors assess with good interexaminer reliability.
With experience and frequent exposure to these methods of assessment, the results and feedback obtained could add value to the learner's experience. Students were recruited by distributing information regarding the study a week prior to the data collection sessions, and written consent forms were completed on the day of the data collection. Students consented to participate and for publication.

Competing interests and Funding
There are no con icts of interests and no funding was received for this pilot study.
Authors' contributions LAM designed the study, collected the data, analysed the data and wrote up the paper. ABN and JF supervised the study, assisted in statistical analyses and write up.