At the time of the study, 99 emergency medicine residency training programs were authorized by the Ministry of Health's Board of Medical Specialties to provide emergency medicine specialty training in our country. Undergraduate medical education was also provided in 71 of these institutions. During this period, the number of emergency medicine educators who met the inclusion criteria was about 400. The questionnaire was shared through the national emergency medicine specialty association, through social media groups, and within clinics. Five of the 95 responses did not agree to participate at the beginning of the questionnaire. Thus the data of 90 participants were included in the analysis.
Demographical data
The median age of the participants was found to be 39 years (min=27; max=64). Sixty-three percent of respondents (n=57) were male, 36.7% (n=33) were women. In terms of academic rank distributions were as followed; the specialists (33.3%; n=30); assistant professor (21.1%; n=19); associate professor (22.2%; n=20); professor (12.2%; n=11) and lecturer (11.1%; n=10). In terms of hospitals' distribution, the highest group consisted of educators working in hospitals affiliated with state universities (44.4%; n=40), followed by the Ministry of Health's Training and Research Hospitals (35.6%; n=32). From the perspective of experience as an educator (median=8 years;1-32) and experience in emergency medicine specialty (median=6 years;1-28), 47.3% (n = 43) of the educators were providing emergency medicine training regularly and periodically.
Computer Skill Levels
In terms of computer use, 51.1% (n=46) participants are at a good level; 21 of them (23.3%) are very good; 21 participants (23.3%) stated that they used a medium-level computer. Only 2 participants stated that they were at a baseline level on computer use.
Distance Education Experiences:
94.4% of the participants stated that they have been teaching by distance education in their clinics. Distance education was not provided in five centers. It was determined that the delivery of emergency medicine residency training through distance education started in 7-11 months with a rate of 56.7% and in 0-6 months with a rate of 36.7%. The target audience of emergency medicine education was mostly emergency medicine residents (83.3%; n=75), emergency medicine specialists (44.4%; n=40), later medical students, and other health personnel. It was determined that 75.6% of the distance learning experience was also in congresses/symposiums and conference activities and 65.6% at the emergency department lectures. The most frequently used distance education platforms were Zoom Client for Meetings (77.8%) and Microsoft Teams (41.1%).
E-Assessment Experiences:
The proportion of participants who experienced an e-assessment was 47.8% (n = 43). While the rate of taking part in the distance exams was 77.4% (n=14), it was found that these educators mostly worked in preparing questions (58.1%; n=25) and evaluating the results (30.2%; n=13). The number of those with supervision experience was found to be 6 (14%).
It was determined that the distance exams were mainly used for medical school students' clerkship exams (83.7%; n=36), 34.9% were used for seniority exams of emergency medicine students, and 20.9% were used for the final examination in emergency medicine residency training. It was determined that the online (53.5%) and offline (34.9%) tests with multiple-choice questions were the most frequently used remote exam type in clinics. It was observed that the examination duration was mainly between 0-60 minutes (83.7%; n=36). 95.3% of the exams were made accessible from anywhere.
Platforms, where examination evaluations (grading) mainly were calculated automatically by the e-assessment platform (74.4%), were at the forefront. 69.8% of the participants with remote test experience stated that audio and video recordings were not taken during the exams, 32.6% stated that only video recordings were taken, and 18.6% of them stated that only audio recordings were taken. The rate of invigilation in online theoretical exams was 44.2% (n=19). The rate of using a question bank/pool in specialty training in medicine was 48.8%, and all participants (n=21) using this pool could also use this pool for remote examinations.
Advantages
Participants believe that the most significant advantages of taking the exams are as follows:
- Accessibility of exams from anywhere (n=73; 81.1%)
- Savings on paper / labor / material (n=67; 74.4%)
- Collection of exam data / statistics (n=57; 63.3%)
- Applicability of the exam to more than one person at the same time (n=56; 62.2%)
- Quick feedback (n=52; 57.8%)
- Repeatability (n=39; 43.3%)
- Applicability of the standard exam format quickly (n=36; 40%)
- Student development follow-up (n = 29; 32.2%)
- Availability of audio-visual invigilation (n = 22; 24.4%)
The participants thought that 72.2% (n=65) of multiple-choice questions and 53.3% (n=48) of multiple true/false question tests were the most suitable exam type for the distance exam. According to the purposes of the exams, the participants believed that the types of summative assessment (56.7%), diagnostic assessment (made at the entrance to the program) (50%), and formative assessment (47.8%) types were more suitable for holding remote examinations.
Concerns
Those who thought that an effective assessment and evaluation could not be done with e-assessment were 13.3% (n=12), 48.9% (n=44) who thought it could be done partially, and 37.8% (n=34) who thought it could be done absolutely.
Situations that the participants perceived as the greatest deficiency were students’ ability to get help from others / cheat during exam (n=67; 74.4%), problems with internet connection / technological infrastructure (n=65; 72.2%), problems in measuring procedural skills (n=57; 63.3%) and ethical problems (n=56; 62.2%). Most of the participants had the opinion of taking deterrent measures against security breaches (n=56; 62.2%); maximizing the security of the question pools (n=54; 60%); and taking audio-visual recordings (n=50; 55.6%).
Security and Ethical Concerns
The rate of ethical concern about video and audio recording about remote exams were found to be 48.9%. The rate of those who were not concerned about this was 26.9%, and those who were partially concerned were 24.4%. Regarding screen sharing, 44.4% of the participants have ethical concerns; 30% were not concerned about this issue, and 25.6% were partially concerned. There was no statistically significant difference in ethical anxiety between those with and without e-assessment experience (p=0.914).
Sharing user codes / passwords (n=59; 65.6%), system being open source / vulnerable (n=58; 64.4%), insufficient verification (not identifying the correctness of IP, MAC addresses) (n=53; 58.9%), security vulnerabilities related to the e-assessment application program (n=59; 65.6%), programs that can share background information/screen (n=60; 66.7%), camera working in the background / voice access programs (n=60; 66.7%) were the most common security concerns.
Age, experience in emergency medicine was not significantly associated with ethical concerns on video and voice recording or screen sharing (p>0.05).
Need for Education
Although there was no statistically significant difference between those with and without e-assessment experience (p>0.05), the participants in both groups thought themselves, students, other colleagues, and administrators needed to receive training on e-assessment.
Factors affecting the Self-Confidence, Self-Efficacy and Attitude Scores
No statistically significant correlation was found between age and self-confidence, self-efficacy, and attitude scores (p>0.05) (Table3). There was no significant difference among genders in terms of attitude (p=0.117), self-confidence (p=0.052), and self-efficacy (p=0.224).There was no statistically significant difference between the academic ranks in terms of scores (self-confidence = 0.170; self-efficacy = 0.772; attitude = 0.respectively).There was no statistically significant relationship found between scores and experience as a specialist and trainer (p>0.05) (Table 3). No significant difference was found among the institutions regarding scores (self-confidence = 0.158; self-efficacy = 0.169; attitude = 0.828). While Age, gender, institution, experience groups did not differ regarding scores.
Table 3. The Correlation of demographical variables with self-confidence, self-efficacy and attitude scores
Variable
|
Self-confidence
|
Self-efficacy
|
Attitude
|
|
r
|
p
|
r
|
p
|
r
|
p
|
Age
|
|
0.325
|
|
0.871
|
|
0.930
|
Gender
|
|
0.052
|
|
0.226
|
|
0.118
|
Experience as EM Specialist
|
|
0.342
|
|
0.764
|
|
0.850
|
Experience as EM educator
|
|
0.112
|
|
0.603
|
|
0.831
|
Experience of E-learning
|
|
0.093
|
|
0.087
|
|
0.147
|
Lectures/semester
|
|
0.060
|
|
0.449
|
|
0.183
|
Computer Self Efficacy
|
0.403
|
0.000
|
0.463
|
0.000
|
0.324
|
0.002
|
Self-confidence
|
1.000
|
|
0.000
|
0.741
|
0.003
|
0.310
|
Self-efficacy
|
0.000
|
0.741
|
1.000
|
|
0.000
|
0.433
|
Attitude
|
0.003
|
0.310
|
0.000
|
0.433
|
1.000
|
|
Abbreviation: EM = Emergency Medicine
*Spearman’s correlation is significant at the 0.01 level
There was a statistically significant difference between computer use skills and scores, especially in terms of self-confidence (p=0.02) and self-efficacy (p=0.01), but no significant difference was found in terms of attitude (p=0.877). Medium, good, and very good levels are statistically higher than baseline levels regarding self-confidence and self-efficacy. There was no difference between medium and good levels. However, very good levels are showed higher self-confidence and self-efficacy scores than medium levels. Also, no statistically significant difference was seen between very good and good levels of computer use regarding self-confidence and self-efficacy. Computer skills were found to affect self-confidence and self-efficacy for e-assessments.
In terms of computer self-efficacy, the median level of problem-solving without technical support during use was 5. There was a correlation between the scores and computer self-efficacy (p<0.05) (Table3). Computer self-efficacy was poorly correlated with attitude (rho=0.324) but positively and significantly correlated with self-efficacy(rho=0.463) and self-confidence (rho=0.403).
There was a significant difference in self-confidence (p=0.000) and self-efficacy (p=0.001) between those with and without e-assessment experience. Self-confidence and self-efficacy of those with e-assessment experience were found to be higher than non-experienced participants. Experience of e-assessment made a statistically significant difference on self-confidence and self-efficacy.
Self-confidence (p=0.005), self-efficacy (p=0.001), and attitude (p=0.000) were found to be statistically significantly higher in those who thought e-assessment is effective. Although no significant difference between gender (p=0.518) or academic rank (p=327) groups, perception of effectivity showed a positive relationship between the increase in self-confidence, self-efficacy, and attitude of e-assessment and the thought that it was a practical assessment and evaluation.
The participants who do not have ethical concerns on video and voice recordings have significantly higher self-efficacy scores (p=0.002). Screen sharing concerns also have a statistically significant association with self-efficacy (p=0.028) and self-confidence (p=0.027), in which participants without concerns on screen sharing were more self-efficient and self-confident.
The summary of the results can be visualized in Figure 1.