Perception of The Use of The DOPS In Occupational Therapy PGY Training In Taiwan: The Perspectives of Trainees And Supervisors

Background: Professional competence of young occupational therapists is enhanced through on-site resources, hands-on training and outcome-based education. The Direct Observation of Procedural Skills (DOPS) is a work-based assessment to evaluate professional knowledge, skills and attitude in occupational therapy (OT) clinical training. The perspectives of OT �eldwork teachers and trainees regarding the DOPS are important, but rarely investigated. This study aimed to investigate the perspectives of OT educators and trainees in using DOPS and their discrepancy for OT post-graduate year (PGY) training. Methods: An online survey was distributed to OT educators and trainees who participated in OT PGY training. The survey comprised �ve sections: demographic information, the practicality in using the DOPS in clinical settings, the ease of rating the DOPS, and advantages and disadvantages of the DOPS. The responses were based on a 5-point Likert scale. A score of 4 or 5 indicated that the responder agreed/simple or strongly agreed/very simple with the statement. A total of 86 and 41 surveys from OT teachers and trainees were returned respectively, and 64 and 30 from OT teachers and trainees who used DOPS were analyzed. Results: Over 70 percent of OT educators and trainees agreed that clinical resources (i.e., time, personnel support and cooperation, feedback and training) were su�cient when using the DOPS. Most OT educators and trainees agreed that DOPS matched with OT training goals, bene�ted the OT competence training and had a fair, objective and consistent scoring system. A signi�cant higher percentage of OT trainees felt stressful in DOPS assessment than that of trainers. Signi�cant higher percentages of OT trainees strongly agreed that DOPS was a practical and appropriate assessment for OT competence training, and had a fair, objective and consistent scoring system than those of OT teachers. There were no signi�cant differences between teachers and trainees regarding easiness of rating DOPS items. Conclusion: Most OT educators and trainees agreed that DOPS was a practical and appropriate assessment for the OT PGY training.

management ability; (2) patient-centered, comprehensive and holistic perspectives and skills; (3) professional ethical reasoning and communication skills; and (4) the ability to work in a team [7][8].
Assessment of clinical competence is a signi cant part of the training for young occupational therapists.Objective and systematic assessment allows both supervisors and trainees to be aware of the training objectives and monitor the progress.The assessment of practical performance in clinical conditions (does) is ranked as the highest level in Miller's hierarchical model for the assessment of clinical competence [9].Direct Observation of Procedural Skills (DOPS) is a workplace-based assessment tool speci cally designed to assess practical skills; it requires the assessor to directly observe a trainee conducting a procedure, document evaluation in a checklist and give feedbacks to the trainee based on objective ndings [10].Several studies have reported supervisor's and trainee's experiences with the DOPS in health professional training such as medical, nursing and veterinary students and showed that both examiners and examinees were very satis ed with the DOPS [10][11].However, the feasibility and user experience of the DOPS for OT PGY training has not been examined.We investigated the perception of OT supervisors and trainees on the DOPS in Taiwan for PGY training.Therefore, the aims of our study were to investigate (1) the experience of using the DOPS in clinical settings; (2) the ease of rating clinical performance with DOPS and (3) advantages and disadvantages of the DOPS for OT PGY training.

Procedures
Data were collected from January 2017 to December 2018.Based on the database of the Ministry of Health and Welfare, Taiwan, OT chiefs in charge of OT PGY training programs in 2017 were invited to participate in this study.The online questionnaire was sent by e-mail along with regular postal mail to participants.Participant's signed informed consent to involve in this study, and their data coding and analysis were anonymized.

Instrument
This questionnaire was developed by two senior occupational therapists (Y.C. Liu and P. C. Huang), based on the research questions and literature of medical education with work-based assessment in 2015 [11][12][13][14].Five experts in OT PGY training were invited to assess and discussed the questionnaire in a focus group and revised this questionnaire accordingly.The second group of another ve external experts in OT eldwork education and research were invited to revise and nalized the questionnaire.
The nal questionnaire had four sections.The rst section included basic sociodemographic information of the participants, including job position, age, gender, highest educational level, teaching/working years, specialty area, status of hospital, and service location.The second section included 21 items related to experiences with using DOPS (i.e., practicality, subjective opinion, cover domain and scoring system) and rated on a 5-point Likert scale.The third section assess trainer's and trainee's perception of ease of using the DOPS in rating the clinical performance of PGY OT trainees with a 5-point Likert scale.The forth part included the advantages and disadvantages of the DOPS for the evaluation of OT PGY trainees.

Statistical Analyses
Demographic characteristics and distribution of participants' responses in perception of DOPS were presented with descriptive statistics (i.e., frequency, percentage, mean and standard deviation).Comparisons between supervisors and trainees in terms of experience of using DOPS, ease of rating the DOPS, and advantages and disadvantages of DOPS were assessed with The χ 2 and t tests.SPSS 18.0 was utilized to analyze data (SPSS, Chicago, IL, USA).

Results
Completed questionnaires were returned by 86 supervisors and 41 trainees of the 95 hospitals (individuals) who received the invitational letter and survey by e-mails and regular mail for a 90.5% return rate.Data from 64 supervisors and 30 trainees who had used the DOPS and completed surveys were included in the analysis.The demographic characteristics of the participants are presented in Table 1.Signi cantly more female, being younger and lower levels of education were noted in the trainee group than in the supervisor group.There were no differences in terms of specialty areas and classi cation of hospital in the two groups.
Regarding experience with DOPS, over 70% of trainees (range: 72.4% to 93.3%) and supervisors (range: 71.3% to 85.9%) reported "strongly agree" or "agree" with most items except for teacher's rating of su cient teachers (67.2%), assessing clinical reasoning ability (62.6%), and consistency in scoring standard (61.0%) and both teacher's and trainee's rating of stressfulness in assessment (17.2% and 41.4% respectively) and need further training (54.7% and 55.1% respectively).In addition, less than 10% of trainees (range: 0 to 6.8%) and supervisors (range: 0 to 9.4%) reported "strongly disagree" or "disagree" with most items except for teacher's rating of need further training (20.6%) and both trainee's and teacher's rating of stressfulness in assessment (10.3% and 23.5% respectively).The use experience of DOPS of the trainees and supervisors and their comparisons are presented in Table 2.
Results of X 2 tests indicated there were signi cant differences between trainees and teachers on most items of experiences of using DOPS for OT PGY training except for items of low cost, better than traditional teaching assessment, fairness, objectiveness, and consistency in scoring standard.Signi cant higher percentages of "strongly agree" were noted in trainee's perception (range: 20.7% to 41.4%) than in supervisor's perception (range: 1.6% to 12.5%).
Most respondents reported "simple" (trainee range: 10.7% to 34.5%; teacher range: 23.0% to 35.0%) or "neutral" (trainee range: 17.2% to 53.6%; teacher range: 27.9% to 47.5%) in the easiness of rating most of the DOPS in clinical settings.Results of X 2 tests indicated there were no signi cant differences between trainees and teachers on all items of easiness of rating DOPS for OT PGY training program.The easiness of rating DOPS as reported by the trainees and supervisors and their comparisons are presented in Table 3.
The advantages of the DOPS assessment tool were ranked as follows: direct observation of clinical performance (trainee: 96.6%, supervisor: 96.8%), immediacy of feedbacks (trainee: 75.0%, supervisor: 76.2%), exibility (trainee: 75.9%, supervisor: 76.2%), nishing assessment in a short period of time (trainee: 69.0%, supervisor: 69.8%), no additional funding or resource burden (trainee: 62.1, supervisor: 55.6%), and no space restriction (trainee: 58.6%, supervisor: 55.6%).Having a valid and reliable scoring system may be an issue for DOPS in the assessment of clinical competence of OT PGY training because over 75% of respondents reported subjectivity as one of the disadvantages.The results of X 2 tests found no differences on perceptions of advantages and disadvantages of DOPS between trainees and supervisors.Table 4 presents the comparison of advantages and disadvantages between trainees and supervisors.

Discussion
To the best of our knowledge, this is one of the few studies to examine the feasibility of DOPS and explore the use experience, advantages and disadvantages for OT PGY training programs by including both perspectives of trainees and supervisors.The results found that the DOPS tool was a practical measurement tool to assess the competence of OT PGY trainees.Time for implementing DOPS and providing feedbacks is a signi cant factor for feasibility and acceptability [11].Some studies [15][16] found lack of time or being seen as an additional workload in administrating DOPS and giving feedbacks in physician training.This may re ect busy clinical hours of the physicians and time constraints for using DOPS in the clinical training of medical doctors.In contrast to ndings of the above studies [11,[15][16], about 90% of the respondents in this study spent at least 11 minutes for rating DOPS and half of the respondents reported to have at least 6-10 minutes for feedbacks.Most of the trainees (86.7% agree or strongly agree) and supervisors (75% agree or strongly agree) agreed time for implementation of DOPS was su cient for OT PCY training.In addition, over 75% trainees and supervisors agree that immediate feedback was one of the advantages of using DOPS in OT PGY training.
Training of rating DOPS is another signi cant issue for consideration in the assessment for OT PGY training.Half of the trainees (55.1%) and supervisors (54.7%) claimed that they need further training in using DOPS.Proper implementation is one of the signi cant factor related to the education impacts of administrating DOPS [17] and relies on su cient training of using assessment tools such as DOPS.About 40% of supervisors (35.9% neural, 1.6%disagree, and 1.6% strongly disagree) did not support consistency in scoring standard of rating DOPS in this study.Inter-case variation may affect the reliability [11].Training is needed not only on the procedure of administrating DOPS, but also on scoring training on different patients or conditions.In addition, a study suggested rating of trainees' performance by at least three supervisors observing at least two procedures each to achieve good reliability [11].
Consistent with previous studies [10][11] that some trainees (41.4% agree or strongly agree) felt that taking DOPS is a stressful experience.However, trainee's satisfaction with DOPS is high and even higher than the supervisor's.It is possible that attending a work-based examination such as DOPS is stressful, but the trainees appreciate this experience as learning and positive though stressful experience.

Conclusion
Most OT educators and trainees agreed that DOPS is a practical and appropriate assessment for the OT PGY training.Su cient time for rating DOPS and feedback about performance are essential to learning effects.Training in using DOPS is necessary and should include rating criteria with different conditions and patients with same procedures.

List Of Abbreviations
OT: occupational therapy, DOPS: Direct Observation of Procedural Skills PGY: post-graduate year

Declarations Ethics approval and consent to participate
The study protocol has been reviewed and approved by the Ethical Committees for Human Research of the National Cheng Kung University Hospital in Taiwan.The study procedures, risks, and bene ts will be explained to all participants, and participants will sign an informed consent before enrolling in this study.Patients will be participating in this study as volunteers and are free to leave the study without any reason.All experiments were performed in accordance with relevant guidelines and regulations.

Table 2 .
Comparison between teachers and trainees in the practicality in using the DOPS in clinical settings

Table 3 .
Comparison between teachers and trainees in the easiness of rating the DOPS in clinical settings

Table 4 .
Advantages and disadvantages of DOPS reported by trainees and supervisors