COVID19 pandemic challenged undergraduate medical education where all in-person teaching and learning activities were disrupted. The healthcare systems were overwhelmed and faced shortages of frontline workers including doctors and nurses. It became imperative to continue the UME courses to ensure the timely graduation of the medical students. Given that, we went through a well-organized and comprehensive implementation process of the ROA program which aimed at transforming all in-person summative curricular assessments into remote web-based assessments during the lockdown. Our findings and first-of-its-kind experience in a developing country’s UME system gives a larger perspective into the feasibility of making such a transition from in-person learning to remote online; a system on which the future of medical education heavily relies.(9)
The overall satisfaction with the ROAs was lower among the preclinical students as compared to the clinical students. This was an important finding that led us to focus more on the preclinical students who are in the initial phase of their medical studies. Additional teachers from the faculty and experts from the IT department were allocated to the preclinical year classes for time to time troubleshooting and guidance.(10) After the first mock assessment, several technical aspects and problems were identified that were causing the students to experience unsuccessful attempts at the exam, problems completing them or interruptions during the remote online exams.
Incompatible devices, power outages due to load shedding, poor internet speed and connectivity were the major problems that were identified. One of the major problems is the lack of uninterrupted power supply and adequate internet connectivity which are the backbone of any online educational activity or assessment.(11) These problems are common and prevalent in the developing world with limited resources.(12)(13) Lack of an alternate source of the internet was identified as a major impediment to smooth exam completion and resulted in frequent interruptions encountered by the students. Our regression analysis after the first feedback also showed that students’ dissatisfaction with ROA was significantly associated with the complexity of the exam interface as the students found it difficult to access and attempt their assessments online. Similarly, dissatisfied students found the allocated exam time insufficient which was mainly because they were given SEQ type questions requiring typing in of answers through keyboards and pads which they never practiced before.
The IT experts made relevant changes in the exam interface following the preferences shared by the students (like adding a Wifi button that allowed them to change their internet connection during the exam). Also, we split an entire exam into three short components, each of which can be attempted and submitted separately. This allowed the students to take small breaks in between and safely submit their exam components once they were complete. This prevented the whole exam to be jeopardized due to any interruption like a power outage or internet breakdown. This also led us to formulate a student-friendly and lean redressal policy for those who experienced technical issues and interruptions during their exams. Such students were allowed to re-take a new exam later on but only in that component that was affected during their primary exam. Also, a few minutes were added to the total allocated time to account for the time consumed in logging in and accessing the exam.
The second training activity comprised an online tutorial session to demonstrate the process of attempting and submitting the exam by teachers and IT experts. These sessions were conducted with a smaller group of students and were followed by a mini-mock exam. The students were then asked to share their concerns and issues which were dealt with and resolved at the spot. This led to the highest satisfaction afterward as all the problems that were previously faced by the students were rectified. Our findings also suggest that having an alternate source of internet other than the 3G or 4G mobile data leads to higher satisfaction among the students with online assessments.(14) These are important considerations to make online assessments successful in developing countries.
After the first online summative mid-term exam, more than forty-five percent of the students showed dissatisfaction which was higher than the second mock exam feedback. This indicates that the students had concerns with taking summative assessments online which were to be counted in their final GPA grading calculation. Further exploration into the 3rd feedback showed that the females and preclinical students were more likely to be dissatisfied which reiterates the findings of earlier feedback. Females in our cultural settings are less computer-friendly and are inexperienced with the technology. This calls for attention towards their training and guidance in computer technology.(15) The grades scored in the mid-term exams were not associated with the satisfaction level which authenticates our findings as the scores achieved in examinations can be a potential confounder to satisfaction with the exam system.(16) The students, while responding to the educational impact and usefulness (EIU) questionnaire, did not agree that these online assessments were better than traditional online exams, by large. However, the majority of the students agreed to the fact that these ROAs are reliable to formulate their GPA and that this assessment program motivated them to study for the exam and prepare their curriculum.(17)
Collectively, these findings suggest that students are satisfied with the ROAs as far as these are formative. This also highlights the importance of designing formative assessments in such a way as to promote learning throughout the year. Overall, this ROA program in our settings achieved its goal by a continuation of the remote learning and assessment process through a combined effort of the technical staff, teaching faculty and the students. This yielded productive outcomes for undergraduate medical education which thrived remotely during the period of complete social isolation and lockdown. This implementation process and the findings from our students’ feedback provide a fertile ground for designing and conducting advanced remote online assessments in the future.
This study is limited by the availability of data from the first phase of implementation which tested a remote online assessment system without live online proctoring and invigilation. Our second phase of implementation included online proctoring using the examinee’s mobile phone camera, however, the data regarding satisfaction with the ROA after incorporating the online proctoring was not available at the time of this study. Also a broad horizon geographically and inclusion of multiple institutions can provide us with even more substantial results that can help enhance the virtual learning and assessment systems.