A Spaced-Repetition Approach to Enhance Medical Student Learning and Engagement in Pharmacology

Background: Pharmacology is a cornerstone of medical education as it underlies safe prescribing practices. However, medical students have reported unease regarding their perceived prociency in clinical pharmacology. Despite the signicant impetus to improve student outcomes, there is little analysis available of the techniques used by medical students to learn, retain and apply pharmacology knowledge. Methods: A mixed methods, student-focused approach was conducted to design and rene specic resources developed to address gaps in pharmacology education. This methodology comprised an anonymised scoping survey, followed by structured focus group interviews. We developed a relevant and time ecient resource to support long-term revision for academic and clinical success. These resources were released to a cohort of 100 graduate preclinical medical students who were invited at the end of year to evaluate the intervention via a subsequent anonymous survey. Results: The scoping survey received 103 complete responses. Surveys and focus group interviews revealed that only 50% of students engage in ongoing revision. The analysis identied in-semester revision of pharmacology as a signicant predictor of strategic and deep learning methods and improved quiz performance (a 5% higher score on average), compared to supercial learning methods. Amongst our cohort, we identied that the evidence-based technique of spaced-repetition was particularly well regarded. Hence, we developed and evaluated a bespoke resource utilising Anki TM , an open-source, spaced-repetition ashcard program. A total of 1208 ashcards spanning 156 distinct classes of drugs with supplementary summary tables, diagrams and explanatory video and summary guides were created. Designed as a strategic revision tool to reinforce learning, evaluation showed students greatly appreciated the “comprehensive” and “well formatted” Anki TM resource that supported existing teaching modalities, with a global rating of 3.8 out of 5. Conclusions: Strategic and personalised techniques for pharmacology learning that assist with in-semester revision and long-term retention are highly valued amongst students for examination preparation and preparedness for practice. Collectively, these results dene a novel approach to identifying and addressing weaknesses in existing teaching methodologies in a manner that is inclusive of, and acceptable to, medical students.


Background
Competence in pharmacology is a crucial outcome in medical education as the discipline provides the rationale for therapeutic interventions and safe prescribing. However, signi cant challenges face medical students, such as developing expertise with the vast and ever-expanding array of medications available on the market, each with unique clinical considerations and interactions (1). Given this, it is unsurprising that many medical students express concerns regarding competence in pharmacology and prescribing, with only 39% of Australian medical graduates perceiving that they were adequately or well prepared in clinical pharmacology (2,3).
Thus, there is signi cant interest in educational reforms to improve student outcomes in pharmacology.
However, it is notable that most approaches have been top-down in nature, focusing on systematic changes in curriculum, teaching and sta ng (4). Hence, there is a gap in the understanding of the approaches and techniques used by medical students to study pharmacology, which could form the basis for targeted interventions to improve learning (5)(6)(7)(8).
Of interest is how students cope with fast-paced integrated medical school curricula and whether support can be provided to encourage more e cient approaches to study (9). One technique of interest is spacedrepetition of ashcards, such as via the open-source and cross-platform digital ashcard program Anki TM , which is known to improve student outcomes (10)(11)(12)(13). This method of learning involves self-testing on ashcards that assess speci c items of knowledge, which are subsequently self-rated for di culty. The program subsequently sets intervals for re-assessment, depending upon prior di culty ratings, establishing a personalised schedule for revision.
Spaced-repetition carries distinct advantages when considered in the context of supporting medical student education (14). Firstly, ashcard revision is a form of retrieval practice that engages students to recall items of information, an act that is known to improve long-term retention ( Figure 1) (15). Secondly, spaced-repetition scheduling is known to improve outcomes via optimising revision to address di cult items whilst minimising repetition of easy concepts. Finally, through dividing complex concepts into discrete items of knowledge to be repeatedly assessed, ashcard revision may encourage a more deliberate and strategic approach to learning that is known to underlie attainment of pro ciency (16).
In this study, we examine the perceptions and approaches of medical students to learning pharmacology, with a focus on identifying and targeting weaknesses through the development of supportive educational resources. In this way, we aim to contribute to a greater understanding of both student perspectives towards learning and possible actionable interventions to improve outcomes.

Methods
Students involved in this study were enrolled in direct or graduate entry Doctor of Medicine (MD) programs delivered by one Australian university. The direct entry degree is a ve-year course, while the graduate entry degree is of four years. Each year level has approximately 350 direct entry students and 100 graduate entry students. All currently enrolled medical students were invited to participate via advertisements on online student noticeboards. Ethics approval was sought from the university's human ethics committee prior to commencement (MURHEC number: 2020-22814-44162).
To better understand medical students' approaches to pharmacology education, an anonymised scoping survey was distributed. An invitation to participate in structured focus group interviews was provided at the end of the survey, with a goal of further elucidating survey ndings and to demonstrate and re ne speci c resources developed. These resources were released to a cohort of graduate entry preclinical medical students who were invited at the end of year to evaluate the intervention via a subsequent anonymous survey ( Figure 2).

Primary survey design and data analysis
To better understand the perceptions and approaches of medical students with respect to pharmacology education, a primary survey was distributed amongst all current students within the participating MD program. The survey primarily consisted of 5-point Likert scales and free text items.
The survey consisted of four distinct sections. The rst section was targeted at identifying key demographic characteristics of the survey participants such as age, sex, year level, entry stream and whether the students were domestic or international. The second section was targeted at characterising student learning approaches, based upon the validated Approach and Study Skills Inventory for Students (ASSIST) survey (17). The ASSIST survey is a 52-item inventory that quanti es the degree to which a student's learning style ts three distinct categories (deep, strategic and surface) that are predictive of academic outcomes (18,19). To reduce the survey length and improve the response rate, a modi ed nine item survey was developed through selecting the nine most predictive items.
The third section of the survey was designed to identify the resources and techniques utilised by students to learn pharmacology. These characteristics were assessed with regard to three distinct time periods of learning; as rst exposure to a topic, as ongoing revision and as revision prior to summative assessments. Finally, the last section of the survey involved a short pharmacology test consisting of 12 multiple-choice questions (Appendix 1). The test was designed to assess core curriculum content with the primary objective of identifying the contribution of demographics to academic outcomes, with the secondary intention of ascertaining any other signi cant predictors of academic success.
Statistical analysis of the data was performed to identify signi cant associations via cross-tabulation, Pearson's chi-squared test, linear regression and Student's t-test using SPSS (version 26) and MS Excel.

Focus group and thematic analysis
Survey respondents were also invited to participate in focus groups to further elicit qualitative information on survey ndings and to evaluate several prospective study resources. Two rounds of focus group interviews with Monash medical students were performed, with a total of ten student participants.

Resource design and implementation
Evaluation of the primary survey identi ed three potential avenues for interventions supporting pharmacology education; Anki TM ashcards, text-based summaries and diagrams. These resources were developed for initial evaluation within focus group interviews and later, a cohort of pre-clinical graduate entry medical students.
Flashcards were designed as cloze sentences, with students self-testing their ability to recall key words or phrases omitted from a pharmacological concept ( Figure 3). Students were then given the opportunity to check their answer and rate the card di culty, which feeds into the spaced-repetition algorithm of Anki TM to effectively schedule future revision based upon need. Anki TM ashcards were created to cover 15 core curriculum topics, spanning 156 distinct classes of drugs, resulting in a total of 1208 ashcards.
Flashcards assessed a diverse range of curriculum material, including basic knowledge such as drug names, mechanisms, indications and adverse drug reactions to more complex knowledge such as clinical practice guidelines.
Traditional text and diagrammatic summaries were also developed to complement the Anki TM resource, allowing students to review the details of the broader drug class and pharmacology topics both within and independent of the Anki TM program.

Follow-up survey design and data analysis
To evaluate the effectiveness of the intervention within the population of pre-clinical graduate entry medical students, a follow-up survey was distributed via advertisements on student forums and noticeboards. Student perceptions of the pharmacology resources were elicited using 5-point Likert scales and free text items, with data subsequently analysed using MS Excel.

Primary survey demographics and results
The survey received 103 complete responses from the population of enrolled students across all years of the medical course. The demographic pro le of the respondents was a median age of 22 years, 69% female, 56% direct entry admission and 81% domestic students. Twenty-six per cent of respondents were in their preclinical years, 44% in their rst clinical year (medical-surgical focus), 11% in their second clinical year (speciality focus) and 20% in their nal year (pre-intern focus) of the course.

Student Learning Preference and Approaches
To better understand the needs of students, learning engagement and techniques were examined with respect to three distinct stages; rst learning a new topic, revision during semester and revision prior to major assessments. Most participants reported signi cant engagement in dedicated pharmacology study to learn new material (95%) and to revise prior to exams (80%). However, only 50% of respondents reported speci c pharmacology revision throughout the teaching year.
To better understand how students learn pharmacology, participants were asked about their usage of several learning techniques of interest (Table 1). With respect to learning new pharmacology material, students showed a signi cant preference towards class activities (63%), text-based notes (84%) and faculty question banks (60%). Relative to the overall level of engagement with studying during semester and prior to exams, students largely reviewed or repeated techniques utilised to learn pharmacology. One notable exception was in the use of ashcards, which had a net increase in usage as in-semester review (30%) and exam revision (63%) as compared to rst exposure learning (35%), despite the lower levels of student engagement at the later time points. Amongst the distinct forms of ashcard revision examined, digital ashcards with spaced-repetition scheduling (e.g. Anki™) were the most popular means of study reported.

Association between revision and predictors of student success
In the test, the mean correct score was 6.25 marks out of 12, with a standard error of 0.27. The lowest correct score was 1, while the highest score was 12. A Chi-squared test of test score over current year level of the course was found to be statistically signi cant ( 2 33 =67.5; p < 0.001) (Fig. 4), with mean test scores increasing across year levels. Performance increased with increased knowledge and clinical learning experience.
Students who reported speci c pharmacology study in-semester were associated with signi cant increases in the deep and strategic approaches to learning items in the modi ed ASSIST survey, relative to students that did not engage in in-semester revision ( Table 2). By contrast, the larger group of students reporting speci c exam revision was associated with signi cant increases in only the strategic approach to learning, as compared to students that did not engage in exam revision. Neither form of revision was associated with a statistically signi cant difference in test scores relative to students that did not engage in revision. Association between the digital spaced-repetition and predictors of student success To identify any potential bene ts of digital spaced-repetition, such as Anki TM , for student success, statistical analysis was performed to compare students who utilised such a revision strategy to those that did not (Table 3). A signi cant association was found between the use of spaced-repetition for insemester review and a strategic approach to learning. A similar association was found with the use of spaced-repetition for exam revision. No statistically signi cant differences were noted in the ASSIST survey results for deep or super cial approaches to learning, nor in the test scores, at any time point of revision. "Flash cards would be fantastic because it is very concise ….
[with] an actual real-world example that would be so much better."

Theme 4: User-friendliness
The main limitation of the Anki TM resource was that it was perceived as "intimidating", particularly for students who have not previously utilised ashcards as a study resource. Introduction of a guide and incorporating ashcards into the study work ow as a post lecture revision tool are potential strategies to enable students to fully appreciate the utility of the Anki TM resource.
Student A3 (preclinical, graduate entry): "I talked to a couple of people who have started using it; but I think they found it quite di cult to go through without going through slides and pre-learning beforehand." Student B4 (clinical, direct-entry): "There are so many cards it is a bit intimidating to try and learn them." Student A2 (preclinical, graduate entry): "Write a manual ...how to use each thing, like step by step? ...Written down; so people can access it at any time."

Follow-up survey demographics and results
The follow-up survey received 29 complete responses from a population of approximately 100 enrolled preclinical graduate entry students to whom the pharmacology resources were rolled out. Reported usage of the Anki TM resource was high at 83%, with 52% of students reporting engagement with the resource to complete all scheduled revision and 31% reporting engagement with the Anki TM resource as a traditional question bank for episodic revision. Overall, the Anki TM resource was well received, with an average rating of 3.8 out of 5. Furthermore, 66% of participants positively received the scheduling of revision by Anki TM and 76% positively rated the long-term viability of Anki TM as a study technique. Notably, 83% of students positively perceived the bene ts of Anki TM as a study technique for subjects other than pharmacology. Furthermore, the accompanying summary tables and diagrammatic representations within the resource package were positively received by 79% and 83% of students respectively ( Figure 5).

Overall thematic analysis and textual analysis based on open-ended questions in the evaluation survey
(What do you think are the strengths of the Anki TM Pharmacology resource? How could the Anki TM Pharmacology Resource be improved?) mirrored the perspectives expressed in the previous focus groups that students regarded the Anki TM pharmacology resource as a "great" and "comprehensive" "study system" for "continuous revision" of key pharmacology concepts.
In free-text comments students highlighted that the resource is not only immediately valuable for their studies for revision and assessment preparation, but long-term knowledge retention in clinical years and future practice as an "accurate resource to refer back to" that "highlights key concepts that we should know not for just this year, but for later years." However, in creating a comprehensive and detailed resource for preclinical students that incorporates clinically-applied pharmacology, 45% of students expressed sentiments that " the sheer volume of some decks felt overwhelming" and that some decks included information "outside the scope" of the preclinical pharmacology syllabus. Integration of the Anki TM resource with case examples and links to useful resources and videos for certain drugs was another suggestion for further development of the resource.
Additionally, requests for "more visual schematics" to be incorporated in the ashcards echoed baseline survey and focus group discussions.

Discussion
Revision and self-testing are critical acts in the process of learning, aiding students in consolidation of their knowledge and building con dence in practice (10). With the demands of modern medical school curricula, students are placed under signi cant pressure to remain up to date with their course content and thus, have signi cant time barriers impeding revision. Here, we found that whilst 95% of participants engaged in dedicated pharmacology study to learn new content and 80% revised prior to exams, only 50% revised during the semester. Qualitatively, participants reported that such revision was often super cially motivated by summative assessment deadlines, rather than for long-term competency. This is consistent with existing studies that suggest students often adopt less effective shallow strategies for learning such as rereading, highlighting and cramming (21,22). Therefore, interventions focusing on engaging medical students in structured revision possess and active learning provides clear bene ts for student outcomes (23).
Given the time pressures and high workloads of medical students, any such intervention must be timee cient and effective (24). Here, we identi ed a statistically signi cant correlation between the use of Anki™ and a strategic approach to learning. This association may have occurred as a result of engagement with scheduled revision by Anki™, which may underlie a key bene t of this intervention as a means of supporting a more methodical and considerate approach to learning that is known to bene t students in transitioning to, and succeeding within, medical school (25,26). Metacognitive awareness and use of spaced study plausibly supports a more methodical and considerate approach to learning, as re ected in the modi ed ASSIST survey results (26). In such a manner, the use of Anki™ could act to support students in developing and maintaining strategic approaches to learning that are known to improve academic outcomes (27).
Beyond requiring a learning resource to be e cacious, students have expressed concerns regarding barriers to entry to ongoing revision. Whilst focus group participants favourably perceived spacedrepetition, many expressed concerns with the initial time investment in the creation of ashcards that ultimately resulted in lesser usage of the technique. This potentially creates a undesirable situation where students that are struggling with time management are least well equipped to access and engage with more time e cient study techniques.
Based upon these ndings, we postulated that student co-development and implementation of a comprehensive Anki™ pharmacology resource, based upon the pharmacology curriculum would bear several advantages. Firstly, as a developed resource on an open source program, such an intervention would lower the major barrier to entry for students to engage in curriculum-speci c pharmacology revision that is time-e cient and freely accessible (28). Secondly, such a resource could encourage more effective revision involving retrieval practice (22) and the option to study with spaced-repetition scheduling that maximises time spent on challenging concepts to improve outcomes (13,16). Finally, the digital nature of the ashcards allows for ease of editing, collaboration and access, thereby ensuring the sustainability of the resource in the long-term (14).
The extensive Anki™ ashcard deck was subsequently incorporated into the pharmacology program with decks of ashcards released to the participating students following traditional classes as a form of ongoing revision in accordance with Bloom's Taxonomy (29) (Fig. 6). Incorporation of Anki™ into the pedagogy, may provide reassurance for teachers that students engaging with the spaced-repetition tool are periodically reviewing delivered material, thereby enhancing their learning long-term.
Evaluation by the participating preclinical student cohort was largely positive, with use of the spacedrepetition scheduled revision being the predominant mode. Student views were largely congruent with the hypothesized bene ts of the intervention; particularly of the strength of the intervention as a comprehensive, prepared resource covering their pharmacology curriculum. Notably, students largely perceived themselves engaging with the resource long-term and expressed an interest in similar resources developed for other medical course content, which potentially serves as a more accurate re ection of the overall perception of the resource potential. The prospect for long-term engagement in the resource is particularly notable as the very nature of Anki™ -based revision essentially ensures re-exposure to, and retention of, curriculum materials well into the future (22,30).
One particular issue identi ed by students was a perception of excessive breadth and depth within the covered material (3). This may be exacerbated in the case of students that struggle with performing scheduled revision, where reviews may accumulate and result in distress and discouragement. Ultimately, we believe that any prepared resource should strive to be comprehensive to ensure long-term viability and for the sake of academic integrity. However, to address this issue, further support should be offered to students on how to tailor their Anki™ program to their particular needs. Examples could include providing additional ashcard metadata to categorise content based upon academic importance or further instructional material to encourage the personalisation of settings (i.e. limiting daily reviews) and how to omit cards from scheduled reviews (4,31).
This study has limitations. Low response rates for the primary and follow-up surveys may have resulted in a signi cant non-respondent bias. Additionally, the assessment of students from a single medical school may have reduced the external validity of these results. Nevertheless, most ndings are consistent with existing literature regarding medical school learning and the use of spaced-repetition as a technique to improve outcomes (13,16). The replication of this study in other cohorts should be conducted to con rm these ndings.

Conclusion
In this study we described a bottom-up approach to supporting medical students learn pharmacology via the investigation of student learning approaches and techniques and the design of an Anki TM spaced-repetition ashcard resource based upon trends observed in the examined student cohort. This resource was well received by participating students, particularly with regards to addressing a signi cant unmet need for time-e cient strategies regarding ongoing revision of pharmacology. In this way, we believe that this approach may be broadly applied across the curriculum to improve medical student education.

Practice Points
Medical students appear to engage minimally with ongoing in-semester revision as a result of fastpaced integrated curricula.
Assessment of student cohort learning can lead to signi cant insight to guide interventions.
Spaced-repetition ashcard resources may promote strategic approaches to learning and support long-term revision amongst medical students.
Spaced-repetition ashcard resources are well-regarded and well-received amongst medical student cohorts.

Declarations
Ethics approval and consent to participate Ethics approval was sought from the university's human ethics committee prior to commencement (MURHEC number: 2020-22814-44162). Consistent with the university ethic committee's guidelines, students were provided with a comprehensive explanatory statement about the study. Informed consent to participate in focus group interviews was obtained in writing, and for the survey was implied through anonymous participation and completion of the online questionnaire..

Consent for publication
Not Applicable. No private information of participants were retained within the dataset collected.

Availability of data and materials
The datasets generated and/or analysed during the current study are not publicly available due potential breach of privacy and con dentiality of participants but are available from the corresponding author on reasonable request.
No funding was received for this paper. Associate Professor SB is Acting Head of School of Rural Health at Monash University. He has been involved in the education of student health professionals and scientists for more than 30 years. He is the co-author of three Australian textbooks, Fundamentals of Pharmacology (in its ninth edition in Australia and in its second edition in the UK), Psychopharmacology for Health Professionals and Principles of Pathophysiology (now in its second edition). He has published a number of journal articles on health professional education, in particular with respect to pharmacology knowledge.

Figure 1
Comparison of exposures to a knowledge with spaced-repetition and traditional techniques. This diagram illustrates the early focus of traditional learning techniques on amassing practice, which creates an initially strong memory that ultimately decays without further revision. By comparison, spaced-repetition is known to effectively distribute meaningful practice across time to reinforce learning and slow the decay of memory, promoting long-term retention.

Figure 2
Flow-chart of resource development and evaluation. This diagram highlights the stepwise iterative process of resource creation. The initial scoping survey of medical students guided the design, development and implementation of bespoke pharmacology resources with evaluation and feedback processes for quality assurance and continual improvement.

Figure 3
Example design of Anki™ pharmacology ashcards. This view shows the back-end editing view of Anki™ ashcards, demonstrating the use of three distinct clozes within a single note to produce three distinct ashcards that assess distinct items of knowledge. When reviewing, the students will be given the text section with a single cloze left out. Once answered, the program will ip the card to show the answer and the more information section of the ashcard.
Page 19/20 Figure 4 Test scores of the pharmacology test (Appendix 1) across course year levels. Graph data is expressed as mean score + standard error.

Figure 5
Likert scale rating of student-guided learning interventions. This graph demonstrates the proportion of different student responses to a 5 point Likert scale evaluating the Anki™ resource developed and implemented from insights gained via the primary survey. Students additionally evaluated supplemental resources provided alongside the Anki™ resource, which were provided in the more information section of ashcards as a means of quickly reviewing a topic.

Figure 6
Incorporation of AnkiTM Spaced-repetition ashcards into pharmacology teaching articulating with Bloom's Taxonomy framework for learning. This diagram demonstrates how resources such as our intervention that assists with review of materials complements traditional methods of teaching and helps to develop a higher level of learning.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. PharmacologyAppendix1Final1.docx