Respondent Characteristics
Of the 724 students enrolled in the course, 678 (93.6%) completed the pre-course survey, and 681 (94.1%) students completed the post-course survey. The matched dataset used in analysis only included the 645 (89.1%) students who completed the self-assessment on both the pre- and post- course surveys. The matched data set includes 48.4% (312) third-year and 51.6% (333) fourth-year medical students. Demographics of students in the dataset are similar to the demographic characteristics of IUSM students overall: 45.9% (296) of respondents self-identified as female, 51% (329) as male, 0.3% (2) as nonbinary, and 2.8% (18) indicated that they prefer not to answer questions about gender identity. Additionally, 25.3% (163) respondents self-identified as minority students, and 67.1% (433) as white students. Lastly, 7.6% (49) of students chose not to answer demographic questions about race and ethnicity.
Findings
Overall, results show higher student ratings on the post-course survey compared to the pre-course survey across all KSA learning domains. For example, composite scores means for each KSA learning domain show an increase in students’ self-assessment ratings after the course was completed compared to the matched pre-course survey results (Figure 1). These results are both statistically significant (p <.001) and substantial as measured by Cohen’s d. Subsequently, details associated with these results are presented. Students’ learning gains in knowledge associated with COVID-19 are presented first. This is followed by findings associated with students’ learning gains in skills and abilities associated with COVID-19.
Knowledge
Results from a paired-samples t-test show students responded with higher self-assessment ratings about their knowledge of COVID-19 on the post-course survey (M = 4.04, SD = 0.53) compared to the pre-course survey (M = 3.03, SD = 0.62) with a significant mean increase of the knowledge composite score of 1.01, 95% CI [0.95, 1.06], t(644) = 36.44, p <.001, d = 1.43 (Figure 2).
It is notable that the effect size, as measured by Cohen’s d, is large (1.43) indicating that students’ scores on the pre- and post-course self-assessments are substantially different. Individual paired-samples t-test conducted on matched responses for the seven knowledge question items show statistically significant mean increases ranging from 0.65 to 1.36 as well as notable effect sizes (Table 1).
Table 1: Students Pre- and Post-Survey Responses on COVID-19 Knowledge Questions
|
COVID-19 Knowledge
|
Pre-Survey (N = 645)
|
Post-Survey (N = 645)
|
Mean Difference
|
Effect Size
|
|
Students' Self-Assessment Ratings
|
M
|
SD
|
M
|
SD
|
(+/-)
|
d
|
K1
|
I am knowledgeable about virology and immunology as they relate to COVID-19.
|
3.08
|
0.81
|
4.02
|
0.61
|
0.94***
|
1.08
|
K2
|
I am knowledgeable about the pathophysiology of COVID-19.
|
2.88
|
0.82
|
3.98
|
0.63
|
1.10***
|
1.20
|
K3
|
I am knowledgeable about the impact of population health in the context of a pandemic, and in particular for COVID-19.
|
3.39
|
0.83
|
4.18
|
0.63
|
0.79***
|
.90
|
K4
|
I am knowledgeable about the impact of social determinants of health in the context of a pandemic, and in particular for COVID-19.
|
3.57
|
0.87
|
4.22
|
0.63
|
0.65***
|
.71
|
K5
|
I am knowledgeable about the ways individuals and organizations can advocate at the state and national level during epidemics/pandemics.
|
2.81
|
0.90
|
3.96
|
0.66
|
1.15***
|
1.13
|
K6
|
I am knowledgeable about the legal aspects that impact COVID-19 patients, providers, and the community during the pandemic.
|
2.47
|
0.87
|
3.83
|
0.74
|
1.36***
|
1.33
|
K7
|
I am knowledgeable about the issues surrounding utilization and preservation of finite resources that impact patients, providers, and the community during the pandemic.
|
3.03
|
0.87
|
4.07
|
0.63
|
1.04***
|
1.07
|
Knowledge Composite Rating Score
|
3.03
|
0.62
|
4.04
|
0.53
|
1.01***
|
1.43
|
*** p < .001
Skills
Students responded with higher self-assessment ratings about their skills related to COVID-19 on the post-course survey (M= 4.14, SD= 0.57) compared to the pre-course survey (M = 3.59, SD = 0.64). This is a significant mean increase of the skills composite score of .55, 95% CI [0.50, 0.60], t(644) = 20.70 p <.001, d = 0.81 (Figure 3).
The effect size, as measured by Cohen’s d is moderate to large showing that students’ scores on the pre- and post-course surveys are notably different. Results from individual paired-samples t-test conducted on matched responses for the five skills question items also show statistically significant differences as well as moderate to large effect sizes (Table 2). Findings related to the skills associated with COVID-19, as well as the other learning domains, are reviewed further in the discussion section.
Table 2: Students Pre- and Post-Survey Responses on COVID-19 Skills Questions
|
COVID-19 Skills
|
Pre-Survey (N = 645)
|
Post-Survey (N = 645)
|
Mean Difference
|
Effect Size
|
|
Students' Self-Assessment Ratings
|
M
|
SD
|
M
|
SD
|
(+/-)
|
d
|
S1
|
I am confident in my skills to use principles of evidence-based medicine, including biostatistics, to evaluate efficacy of therapeutic interventions for COVID-19 infection.
|
3.37
|
0.94
|
4.05
|
0.69
|
0.68***
|
.72
|
S2
|
I am confident in my skills to analyze the management of epidemics and pandemics historically and in modern medicine.
|
3.14
|
0.90
|
4.09
|
0.67
|
0.95***
|
.96
|
S3
|
I am confident in my skills to identify a research question.
|
3.98
|
0.77
|
4.26
|
0.67
|
0.28***
|
.36
|
S4
|
I am confident in my skills to appraise the quality and credibility of a source and synthesize the information to advance my understanding of pandemic responses.
|
3.84
|
0.77
|
4.14
|
0.66
|
0.30***
|
.36
|
S5
|
I am confident in my skills to implement basic strategies for mental health and wellbeing promotion for providers in the face of a healthcare emergency, and understand their importance to overall health.
|
3.60
|
0.81
|
4.15
|
0.65
|
0.55***
|
.61
|
|
Skills Composite Rating Score
|
3.59
|
0.64
|
4.14
|
0.57
|
0.55***
|
.81
|
*** p < .001
Abilities
Students responded with higher self-assessment ratings about their abilities related to COVID-19 on the post-course survey (M= 4.05, SD= 0.53) compared to the pre-course survey (M = 3.03, SD = 0.64). This is a significant mean increase of the abilities composite score of 1.02, 95% CI [.97, 1.07], t(644) = 36.56, p <.001, d = 1.44 (Figure 4).
Similar to the findings associated with the other learning domains, the large effect size also indicates that the differences in students’ responses are substantial. Results from individual paired-samples t-test conducted on matched responses for the five abilities question items also show statistically significant differences and large effect sizes (Table 3).
Table 3: Students Pre- and Post-Survey Responses on COVID-19 Abilities Questions
|
COVID-19 Abilities
|
Pre-Survey (N = 645)
|
Post- Survey (N = 645)
|
Mean Difference
|
Effect Size
|
|
Students' Self-Assessment Ratings
|
M
|
SD
|
M
|
SD
|
(+/-)
|
d
|
A1
|
I am confident in my ability to recognize the clinical presentation of a patient with COVID-19.
|
3.29
|
0.85
|
4.12
|
0.63
|
0.83***
|
.92
|
A2
|
I am confident in my ability to outline a treatment course for suspected COVID-19 patients.
|
2.52
|
0.89
|
3.82
|
0.75
|
1.30***
|
1.32
|
A3
|
I am confident in my ability to identify at-risk populations for poor outcomes with COVID-19.
|
3.54
|
0.83
|
4.26
|
0.60
|
0.72***
|
.82
|
A4
|
I am confident in my ability to describe disaster medicine principles, including the processes and policies by which community and international agencies interact to coordinate a safe and effective disaster/pandemic response.
|
2.55
|
0.88
|
3.94
|
0.65
|
1.39***
|
1.42
|
A5
|
I am confident in my ability to identify ways of modifying communication strategies based on the context.
|
3.26
|
0.88
|
4.11
|
0.62
|
0.85***
|
.89
|
|
Abilities Composite Rating Score
|
3.03
|
0.64
|
4.05
|
0.53
|
1.02***
|
1.44
|
*** p < .001
Additional Findings
Student Satisfaction
The students in the sample also provided useful feedback about their satisfaction with the course and the virtual delivery of course content using the same five-point Likert scale that was used to assess the KSA learning domains. Results from the post-course survey regarding virtual delivery showed mean ratings of 4.02 (SD = 0.75) and 3.83 (SD = 0.93) for the following statements, respectively: “The technology generally worked well for this virtual course” and “The content was delivered effectively in a virtual format.” Results from the student surveys regarding overall course effectiveness showed mean ratings of 3.70 (SD = 0.996) and 3.69 (SD = 1.02) for the following statements, respectively: “Overall, this course provided an effective learning experience.” and “The virtual delivery of course content did not detract from the overall effectiveness of this class.” Taken together, these results demonstrate that students were generally satisfied with the Fundamentals of COVID-19 course and that they viewed the technology and virtual format positively. This provides further evidence in support of student-led curriculum design and online instruction in medical student education.