Designing, Implementation and Evaluation of an Eportfolio for Assessment of Surgical Technologist Students in Iran: a Mixed Method Study

Background: Building a portfolio is an appropriate method for assessing competency and practice in clinical settings. Portfolio assessment provides opportunities for students to reect on their performance and therefore could have a critical role in observing students' progress and developing competencies. With the emphasis of modern medical education on e-portfolios, the present study was done to design, implement, and evaluate an electronic portfolio to assess surgical technologist students learning in Iran. Methods: This mixed study was conducted in three stages: "designing," "implementation," and "evaluation." The "design" stage included a literature review, the completion and modication of the list of portfolio items, and content approval by experts. The "implementation" stage involved loading the system, holding workshops for students and clinical preceptors. Finally, the "evaluation" stage included assessing preceptors' and students' satisfaction, knowledge, and clinical skills and recording the number of reections. Results: The paired sample t-test result shows that students' knowledge and clinical skills signicantly improved. More than two-thirds of the students and preceptors were thoroughly satised with the eportfolio assessment. On average, each student had more than three reections recorded. Conclusions: Using an e-portfolio assessment satises students and preceptors, improves surgical technologist students' knowledge and skills, and supports their progress toward achieving professional goals; therefore, it is recommended as a comprehensive and cost-effective assessment method.


Background:
Today, education of medical and paramedical students is mainly based on competency-based education (1). In competency-based education, there is an emphasis on educational consequences, professional practice, and assessment methods; therefore, diverse means of evaluation such as portfolios are proposed to appraise learners' competencies and practices (2). However, the term "Portfolio" rst became popular in education in 1990 and was rst de ned as an organized complex of learnt subjects (3). It was broadly used in medical education over the last two decades to develop learners' competency (4). Today, portfolio is known as a tool for gathering and assessing information relevant to learners' development in appointed assignments and de ned competencies (2,5) and it is introduced as a competent method in motivating and guiding the learners towards re ection in assessment, Preceptors, by providing opportunities for re ection, assist students in not only developing their knowledge and skill but also in correcting their practice and acquiring competency (6, 7). Thus, the relation between the students' knowledge and practice is improved (8). Portfolios are available in two formats: paper and electronic. However, for a variety of reasons, including ease of access, prevention of excessive consumption of paper, and environmental concerns, electronic or network-based portfolios are becoming more popular (9). Electronic portfolio is a learning and assessment tool with features such as saving, reconsideration of data related to student's practice over a certain period of time and in a digital environment for a speci c goal. It can present these documents and evidence to one or more audiences. In addition, it can provide bene ts such as increasing motivation in students, sharing contents with evaluators and other students, hyperlinking the data, easily updating contents, easily accessing portfolios for both students and schools, reducing paper consumption, making students reconsider and nally students can recognize their strengths and weaknesses in their academic years (10,11). However, there are aws such as the requirement for an internet connection and computer skills as well as ethical challenges, which in some cases cause students and teacher to be reluctant to use this assessment system(12) Therefore, it is recommended that in addition to emphasizing the several bene ts of this system of assessment on students' learning, this system be used mandatorily in medical science universities. Results show that despite the initial resistance against this system, after a while, learners use it in a much more active, autonomous and dynamic manner, and move towards self-learning with signi cant growth in critical thinking (13).

Methods:
This is a mixed method study, which was designed, implemented and assessed after approval of initial The rst step of this stage was to conduct a review in PubMed, Science Direct, Google scholar, SID, Scopus, Magiran databases between 1990 and 2019 with the keywords of clinical assessment, assessment tools, portfolio, e-portfolio, logbook and studies related to portfolio and e-portfolio were included in the study. Then, items considered in previous studies' portfolios were listed. In the second step, a meeting with the operating room board as an expert panel was held and the prepared list was corrected and completed in accordance with educational goals of the operating room curriculum. Then in the third step, a meeting was held with an expert working for Jame Tabib Electronic System Launching Company and the requested program was delivered to him in order to design and upload it in the eportfolio assessment system. The nal content that approved for uploading included discipline regulations, study resources, training syllabus, logbook, educational objectives of the course, course activities, an interactional space for receiving feedbacks and providing re ection opportunities.

Implementation:
In this stage, the rst step was to upload the e-portfolio assessment system, which included all of the features and contents of the research team (https://tabib.abzums.ac.ir ). The second step was to hold seven educational workshops for preceptors and students to familiarize them with the system and its operation. These workshops were held with 7 preceptors and 56 students in eight groups. During the training classes, verbal informed consent was obtained from the participants. In the end of this stage, the system was used to assess of the students in the rst semester of the 2020.

Page 4/9
Evaluation: In order to evaluate the program, the Kirkpatrick model was used in two stages: reaction and learning (14). For this purpose, Ahmadi et al. survey was used to assess students' satisfaction (15). Also, a standard multiple-choice question (pre-posttest) and checklist compatible with the educational goals were used to assessment of students 'knowledge and skills. In addition, due to importance of re ection in the portfolio assessment, the number of re ections recorded by students was assessed as one the consequences of learning in different elds.

Results:
Participants of this study included 56 students and seven preceptors. Sixteen students (28.5%) were male, while 40 (71.5%) were female. The mean age of the students was 1.74 ± 19.01. The preceptors were all female, with a mean age of 1.37 ± 38.96. Table 1: compares students' knowledge and skill scores and Table 2 :shows students and preceptors' satisfaction levels with the portfolio assessment .
The paired sample t-test results show that the knowledge and the practical skills of the students have signi cantly improved after the portfolio assessment. Results show that more than two-thirds of the preceptors and the students were completely satis ed with the e-portfolio assessment. In an analysis of the survey's open-ended question (would you please explain the advantages and disadvantages of this method of assessment in comparison to other commonly used methods?), instances like resource management, the need for an internet infrastructure, access to information, improvement of interactions, organized assessment, learning combined with assessment, transparency of assessment and opportunities for re ections and independent learning were identi ed. An analysis of the number of recorded re ections shows that, on average, each student had more than three recorded re ections with preceptors' feedbacks during their 16 days of training. Forty-ve percent of these recorded re ections were about surgery techniques, 33% were about surgery-related care, 12% were about observance of aseptic principles and sterilization and 10% were about surgical equipment. Discussion: The ndings of this study indicate that e-portfolios have a positive impact on students' learning in cognitive and psychomotor areas, as both knowledge and clinical skills of students improved after this project was implemented. As a result, Lai C-Y et al. claimed that portfolios aided clinical learning through features such as self-awareness, self-assessment, and self-judgment (16). Furthermore, Vaghee et al.
reported that e-portfolios improved students' academic achievement by involving students in the teaching-learning process (17). Brown demonstrated that portfolios improved students' learning and changed their perceptions of this profession (18). In other studies, portfolios enhanced learning during training in a women's ward (19) and enhanced both quality and the depth of learning in medical students (20). Gomez et al. investigated e-portfolios to assess quantitatively surgical skills. Seventy-nine percent of the students felt they had more access to their professors and at the end of each course, on average, students reached 65% of the de ned general goals and 87% of skill goals (13). In another similar study, this tool increased clinical competency of nurses in all aspects (21). All of the preceding studies are consistent with this study's results. Therefore, it seems that this method can improve clinical skills by increasing interaction between the student and the preceptor, increasing feedbacks and encouraging students to reconsider.
Another outcome measured in this study was satisfaction of students and preceptors with the portfolio assessment. The results show that most of the students and preceptors were satis ed with the e-portfolio for assessment of clinical competency. Therefore, Asadi et al. preferred this method of assessment over other commonly used methods (22) and in other similar studies, students were more satis ed with the portfolio method than the other commonly used methods because this method focuses on students' progress and feedbacks from preceptors (20). According to Ahamdi et al., the total score of satisfaction with e-portfolio assessment was higher than the other common assessments (15). Contrary to these results, Tailor and colleagues believed 60% of medical interns was not bene cial with e-portfolios (10). Also, students in Gillian et al study were mostly dissatis ed and only 22% of them found e-portfolios useful (23). In another study, students reported low motivation and moderate self-con dence when it came to using e-portfolios (24). Different results can be due to initial anxiety in applying this method, incompletion of the prepared system content, unfamiliarity with the system and lack of a su cient supervision. Gulden et al studied 90 e-portfolios and indicated that the received feedbacks were mostly limited to general opinions and lacked speci c comments and recommendations for further actions and also no certain goals were de ned for learners (25). Thus, considered items, preparedness of preceptors and learners, motivation for constant and correct use of this tool all impact its e ciency and students' satisfaction. Mi Yoo and colleagues believe that when evaluators with an appropriate education do the assessments based on certain standards and in a systematic way, the results are more valid (5). The last result, assessment of the number of recorded re ections, showed that, on average, each student had more than three re ection experiences with feedbacks from preceptors. Developing re ective skills is one of the unique features of portfolios that lead to improving clinical skills (26). During the implementation of this project, there was always an emphasis on re ection with the feedback from preceptors. 51.3% of students in a study reported that the re ection was helpful .other studies recommend that using eportfolios with feedbacks could help students in academic achievement (7).

Conclusion:
Regarding the positive impacts of this method of assessment on students' satisfaction, knowledge, and skills, it seems that applying an e-portfolio as one of the novel tools of assessment can be a proper replacement for the present assessment methods by providing opportunities for feedback and re ection. Therefore, this method is recommended in the clinical assessment of surgical technologist students.
Abbreviations: e-portfolio: electronic portfolio Declarations: Ethics approval and consent to participate This project has been accepted by the ethical committee of the Iran National Center for Strategic Research in Medical Education under ethical code #97006. The authors con rm that all methods were performed in accordance with the relevant guidelines and regulations. Researchers obtained verbal informed consent from all participants after giving similar educational information to the participants in the training meetings about the study's objectives. Also, obtaining verbal consent from the participants was mentioned in the initial proposal of this research, and the ethics committee was aware of this.
Consent for publication "Not applicable" Availability of data and material