Measuring Outcomes of Educational Activities with a Novel Virtual Patient Simulation-Gamification Platform

doi:10.21203/rs.3.rs-3142191/v1

Download PDF

Research Article

Measuring Outcomes of Educational Activities with a Novel Virtual Patient Simulation-Gamification Platform

https://doi.org/10.21203/rs.3.rs-3142191/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The goal of continuing medical education (CME) is to help healthcare providers (HCP) improve their knowledge and levels of competency and enhance their professional performance in practice. Despite the long history of the CME and the CME-like programs, proof of their success in demonstrating improved clinical outcomes has been difficult to obtain. Lately, digital encounters of virtual patient simulation (VPS) have disrupted medical education, replacing traditional presentations followed by multiple-choice questions as a means of assessing learning and progress.

Methods

The most current VPS applications can deliver educational activity in an engaging, “video game-style” way, allowing practice in a safe environment, and providing individualized detailed analysis of potential outcomes. We utilized the advanced VPS gamification platform (Edocate Ltd., Ramat Gan, Israel) to analyze the outcomes of educational activities offered in various settings. We analyzed the data collected by the platform in real-time that provide an objective picture of the learner’s performance of physical examination, diagnostic considerations, prescription patterns of various medications, appropriateness of laboratory and imaging investigation, referral patterns, and adherence to guidelines and best practices.

Results

Three virtual patient scenarios (obesity and type 2 diabetes) were offered to the learners with the time interval between the cases varied from 1 day to 6 weeks, 10 days on average, allowing us to assess the durability of the acquisition of knowledge. This CME activity was tested in 126 physicians and 261 advanced practice providers. Data analysis identified gaps in knowledge and competence, adherence to the most recent guidelines, selection and application of auxiliary tests, referral pattern to specialists, as well as prescription patterns and familiarity with medications and their side effects. From the learner’s side, utilization of VPS resulted in improved performance from one case to another. From the program host’s point of view, the advanced VPS platform allowed the development of measurable, detailed insights into what learners have done individually and as a group.

Conclusions

This study demonstrates the effectiveness, advantages, and limitations of the VPS-gamification platform in assessing knowledge gaps, performance, prescription patterns, and clinical competency improvement among learners.

virtual patient simulation

continuing education

objective assessment of gaps in knowledge

improvement in clinical competency

gamification in education

The ultimate goal of continuing medical education (CME) is to help healthcare providers (HCP) improve their knowledge and levels of competency in order to enhance their professional performance in practice (1). Accordingly, CME is designed and administered to facilitate life-long learning among the HCP.

Many other educational programs have structures similar to CME. These include, but not limited to, maintenance of certification exams (MOC), classes and educational seminar in medical and nursing/physician assistant schools, as well as seminars and courses offered by professional societies, hospital systems, and industry partners. These programs serve to increase and maintain the knowledge, competence, and professional performance of the providers and/or students. Despite the long history of the CME (2, 3) and the CME-like programs, proof of their success in demonstrating improved clinical outcome has been difficult to obtain. Even when the answers to a few post-activity questions show an improvement over the pre-activity test, it is not trivial to ascertain how this improvement would result in better clinical performance (4–9).

Because outcome measures are essential to the educational process, there has been a persistent interest in developing tools to assess the effectiveness of the CME and various medical educational programs. To demonstrate the change (hopefully, an improvement) in the provider’s knowledge and competence, objective measures must be applied to analyze learners’ performance. To analyze this change, a certain state of knowledge (or a gap in knowledge) must be assessed before and after the educational intervention. The CME providers have been developing such tools over the years with variable levels of success (5, 6, 7, 10).

During the past several years the world of medical education (both primary and continuing) has been disrupted by digital encounters of virtual simulation that began replacing lectures followed by multiple-choice questions as an assessment of learning and change (11–14). The most current virtual patient simulation (VPS) applications can deliver the educational activity in an engaging, “video game-style” way, combining gamification with VPS This strategy allows the learner to practice in a safe environment while providing individualized detailed analysis of potential outcomes. The latter includes identification of gaps in knowledge and competence, adherence to the most recent guidelines, selection and application of auxiliary tests (e.g., radiology, clinical chemistry, pathology, etc.), referral pattern to specialists, as well as prescription patterns and the selection and familiarity with medications (15, 16).

Gamification refers to the incorporation of game elements into non-game settings (17). In education, the concept of gamification has gained traction during the last decade or so due to the advance of web technologies and mobile applications. When applied to education, gamification implies an approach that increases learners’ motivation and engagement by incorporating game design elements in educational environments (18).

From the learner’s side, utilization of VPS results in improved performance from one case to another. From the program host’s point of view, since all learners address the same virtual cases, the advanced VPS platform allows the development of measurable, detailed insights into what learners have done individually and as a group. The addition of the elements of gamification might further enhance learning behavior and improve the outcomes of the learning processes. In the case of CME, combining VPS with gamification may improve the effectiveness of CME activities.

The goal of this communication is to outline the advantages of the VPS approach to CME by providing examples of analysis of the CME activity performed by 387 learners in 2022. The data sets presented in this manuscript are used to highlight the capability of the VPS-gamification platform rather than to make comparisons between the groups.

A full description of the VPS platform used for the current analysis has been published previously (15, 16). In brief, the mobile and desktop applications of this platform have been used in several CME and other training activities administered by professional organizations, government agencies, and multinational pharmaceutical companies.

In this communication, we wish to share the capability of the advanced VPS-gamification platform (Edocate Ltd., Ramat Gan, Israel) to analyze the outcomes of educational activities offered in various settings. We concentrate specifically on the analytical data collected by the platform in real-time that provide an objective picture of the learner’s performance of physical examination, diagnostic considerations, prescription patterns of various medications, appropriateness of laboratory and imaging investigation, referral patterns, and adherence to guidelines and best practices. More than 10,000 unique learners used this application in various CME and CME-like activities over the past 3 years.

Like in real life, many cases have overlapping or similar medical problems, allowing the learner to demonstrate their improved performance without repeating the same case. Three virtual patient scenarios were offered to the learners:

Patient 1–52-year-old female, with Type-2 diabetes (T2DM), hypertension, hyperlipidemia, and obesity. Body mass index (BMI) 32.7, hemoglobin A1c (HbA1c) 9.2%, treated with metformin, insulin glargine, simvastatin, amlodipine and lisinopril. The patient was recently hospitalized due to severe hypoglycemia.

Patient 2–36-year-old male, T2DM, hypertension, and obesity. BMI 42.1, HbA1c 9.1%, treated with metformin, insulin glargine, insulin lispro, atorvastatin, lisinopril, and hydrochlorothiazide. The patient exhibited poor compliance with medical therapy.

Patient 3–41-year-old female, recently diagnosed T2DM. BMI 42, HbA1c, 8.1%, treated with paroxetine. She has a history of sleep disturbance, depression, and anxiety.

The time interval between the cases varied among the learners from 1 day to 6 weeks, 10 days on average, allowing us to assess the new levels of competency and the durability of the acquisition of knowledge. The analysis of the collected data provides a direct and immediate assessment of the knowledge gap and its improvement as the learner moves from one virtual patient to another.

For illustrative purposes of the value and capabilities of the VPS platform, we present the data from one such activity comparing the performance of physicians and advanced practice providers (APP). This CME activity was utilized by 126 physicians (44 specialists in internal medicine, 37 general practitioners, 6 surgeons, 6 psychiatrists, 4 pediatricians, 4 emergency medicine physicians, and 25 specialists in cardiology, hospital medicine, neurology, endocrinology, gastroenterology, hematology, and nephrology), as well as by 146 nurse practitioners (NPs) and 115 physician assistants (PAs) (a total of 261 advanced practice practitioners (APP)(. The learners addressed the 3 virtual patients with obesity and T2DM whom they followed for up to 8 virtual months.

Identifying knowledge gap.

The knowledge gap is the difference between the current knowledge of a given learner and the optimal state of knowledge. Using the VPS platform allowed us to assess this difference in the performance of physical examination, recording relevant diagnoses, ordering appropriate laboratory and imaging tests, referring patients to specialists as needed, prescribing the most effective medications, and selecting optimal follow-up.

Learner performance analysis revealed that 51.6% (n = 65) of physicians and 55.9% (n = 146) of APP achieved a meaningful reduction of HbA1c (more than 10%) - (Table 1). However, only 19.0% (n = 24) of physicians and 17.2% (n = 45) of APP achieved a level of HbA1c below 7% in their patients (Table 1).

Table 1

Achievements of glycemic control and weight reduction by the learners
	Physicians (n = 126)	APP (n = 261)
Glycemic Control
> 10% HbA1c reduction	51.6%	55.9%
Reach HbA1c < 7%	19.0%	17.2%
Weight Reduction
0–5%	53.2%	54.4%
5–10%	31.0%	35.2%
> 10%	15.1%	10.7%
HbA1c: hemoglobin A1c

Substantial weight reduction (> 10% of the original weight) was achieved by 15.1% (n = 19) of Physicians and 10.7% (n = 28) of APP (Table 1). Approximately one-third of physicians (31.0% n = 39) and APP (35.2% n = 92) achieved weight reduction between 5 and 10%. The remaining physicians (53.2% n = 67) and APP (54.4% n = 142) achieved less than 5% weight reduction in their patients.

While referrals to ophthalmology and foot examination showed a high level of compliance with the guidelines among both physicians and APP, their utilization of lifestyle changes (physical activity and dietary intervention) were inappropriately low (ranging between 61.9% and 74.3%), supporting the notion of frequent disregard of these important elements of management of obesity and T2DM. Finally, only 46.0% (n = 58) of physicians and 61.6% (n = 161) of APP ordered appropriate laboratory tests for their patients (Table 2).

Table 2

Percent of learners achieved pre-determined goals of the activity
	Physicians (n = 126)	APP (n = 261)
Goals	% achieved	% achieved
Foot care	80.2	81.2
Mandatory labs	46.0	61.7
Ophthalmology referral	82.5	81.6
Patient education	84.9	85.1
Meal planning	67.5	74.3
Physical activity	61.9	70.5
Shared Decision Making	90.5	93.5
Sleep study	75.4	79.7

Prescription patterns.

The VPS platform allowed an assessment of prescription patterns of individual learners and groups of learners. The first VPS case introduced a 52-year-old woman with obesity (BMI 32.7) and poorly controlled T2DM (HbA1c of 9.2%) treated with metformin and insulin glargine.

The selection of the first add-on medications at the initial visit is depicted in Fig. 1: 10.3% (n = 13) of physicians prescribed weight management medications to this patient, 59.5% (n = 75) prescribed glucagon-like peptide-1 receptor agonists (GLP-1 RA), 22.2% (n = 28) sodium/glucose cotransporter-2 inhibitors (SGLT2i), and 7.1% (n = 9) short-acting insulin. Among APP these percentages were similar but more learners in this group prescribed dipeptidyl peptidase-4 inhibitors (DPP4i) and sulfonylureas (2.1% n = 6 and 6.4% n = 17 respectively). Interestingly, 22 learners used a combination of two medications added simultaneously at the first encounter (not shown).

For greater granularity, the platform allowed us to examine individual preparations of medications prescribed by learners. The example of the GLP-1 RA preparations prescribed is shown in Fig. 2.

Outcome measures and durability of acquisition of knowledge.

At the end of each case, the learners received detailed feedback on goals achieved and missed (the goals were predefined by the faculty who created the case). The feedback contained short explanations for every goal with references to appropriate guidelines and available literature. Figure 3 outlines both the improvement in performance and the durability of the acquisition of knowledge from case to case by all learners, as the subsequent cases were addressed by the learners on average 10 days after the first case.

CME was initiated in the US around 1920–1930 (2) and adopted by the American Academy of General Practice in 1947 (1). The American Medical Association presently defines CME as “educational activities that serve to maintain, develop, or increase the knowledge, skills, performance, and relationships a physician uses to provide services for patients, the public, or the profession” (3).

Even though it is generally accepted, albeit with the low quality of the evidence, that CME appears to be effective at the acquisition and retention of knowledge, attitudes, and skills (4–7), the remaining fundamental question about CME and CME-like activities is whether they improve HCP performance in real-life practice and patient health outcomes. The answer to this question awaits an introduction of objective measures of assessment of closure of gaps in knowledge and improvement in competency of learners.

Direct and objective measurement of learning remains to be an imprecise science with the best estimates derived from pre- and post-activity testing. Thus, the effectiveness of the CME activity has been traditionally evaluated by comparing the performance of a learner on pre- and post-activity exams. This approach is not totally reliable in translating exam answers into better practical performance (5, 6, 7). In fact, it has been frustratingly difficult to measure objectively the relationship between teaching, learning, clinical competence, and patient outcome (8, 19–21). In the absence of these direct and objective data, the analysis frequently relied upon self-reported gains in knowledge and behavior change as a result of participation in the CME activity (9, 10, 22). However, subjective evaluation based on self-reported knowledge gain is even less reliable (23, 24).

Even such frequently used and quoted Moore’s Seven Levels of CME outcome measures and Miller pyramid assessment are based largely on subjective self-reported observations (5, 6, 7, 24). Objective evaluation of improved performance such as chart reviews and direct observations of the learner’s skill in a practice setting is much more cumbersome and rarely if ever employed outside the research setting.

In the past several years, simulation and other interactive techniques have been introduced into the CME world (11–14). These techniques not only offer an effective approach to learning, but they can also improve the assessment of the effectiveness of digital learning.

VPS is defined as an interactive computer or mobile application simulation of real-life scenarios applied for the training of health professionals. Simulation allows learners to practice skills and improve their critical thinking and level of competency without any risk to a patient (25). VPS has been successfully employed in teaching medical and nursing students, residents, and practicing physicians (26–28).

In 2019 Kononowicz and colleagues (13) have performed a systematic review of the effectiveness of VPS in professional education following the Cochrane methodology, searching databases from 1990 to September 2018. They analyzed a total of 51 trials with 4696 participants comparing the effectiveness of VPS to traditional education. The authors found similar results for knowledge and favored the VPS approach for skills. They concluded that VPS can be more effective in improving skills and at least as effective as traditional education in improving knowledge.

Herein we report the constructive and effective utilization of a new generation VPS platform to measure and compare objectively a) a multitude of learning and performance parameters; b) performance of an individual learner relative to their peers; c) performance of an individual learner relative to a given group of learners; d) relative performance of one group of learners to another; e) improvement in knowledge/competency of individual learners or groups of learners; and f) durability of learning.

In addition, the analysis of the VPS performance reveals prescription patterns of individual learners and those of groups of learners allowing an assessment of their familiarity with the current guidelines and the array of medications.

The analysis brings to light not only the selection of medications but also the timing of drug initiation and termination, as well as the patterns of combination therapy. Analysis of learners’ performance in selecting medications also determines their knowledge of and adherence to national guidelines and local formularies as appropriate. Furthermore, employing VPS methodology allows to demonstrate how familiar the learners are with the side effects of various medications and their level of knowledge as to how to prevent or deal with these side effects. Finally, VPS analysis allows the provider of the activity to define objectively learners’ prescribing experience with potential prescription errors (29).

Traditional pre- and post-activity testing may still be employed along with the VPS-based educational activity, but in these cases, the data from the latter drive an objective assessment of what the learners have done while addressing a variety of clinical scenarios.

We believe the success of the described VPS system owes much to its game-like design. Medical education is changing at a fast pace. Both graduate and post-graduate medical education incorporate technology-enhanced active learning and multimedia education tools into their curriculum. Gamified training platforms include educational games, mobile medical apps, and virtual patient scenarios. VPS has been consistently shown to enhance learning results in general. Gamification has the potential to further improve learning, engagement, and cooperation (30). Especially, a combination of VPS and gamification may help with promoting risk-free healthcare decision-making, remote learning, learning analytics, and quick feedback.

The limitations of the VPS-based approach are related primarily to the use of technology. First, developing VPS cases requires specific technical knowledge to translate a “written” case into a virtual simulation with diverse pathways based on self-propelled simulation of disease progression and the effects of treatment.

Second, learners with a familiarity with software (practice with previous VPS or games) have an advantage as this familiarity prevents computer-based errors.

Third, future quality assurance studies are still needed to evaluate whether closures of “gaps in knowledge”, “increase in competence” and “performance improvement” relate to the real-life practice of these learners. As VPS, particularly in combination with gamification, competes and replaces the traditional CME approach, such studies will inevitably be performed.

Based on our experience we report the effectiveness as well as advantages and limitations of the VPS platform in assessing gaps in knowledge, performance and prescription patterns, and improvement in the clinical competency of learners (15, 16).

VPS – virtual patient simulation; HCP - healthcare provider; CME - continuing medical education; NP - nurse practitioner; PA - physician assistant; advanced practice practitioners – APP; type-2 diabetes – T2DM; body mass index – BMI; hemoglobin A1c – HbA1c; glucagon-like peptide-1 receptor agonists - GLP-1 RA; sodium/glucose cotransporter-2 inhibitors - SGLT2i; dipeptidyl peptidase-4 inhibitors - DPP4i; thiazolidinedione - TZD

Ethics approval and consent to participate: The purpose of this observational retrospective study was to assess outcomes of educational activities where the participants had worked on a training module for continuing education, as part of their normal educational practices. The subject's data was anonymous. As such, according to 45 CFR 46.104(d)(1), it did not require IRB, nor the need for informed consent was deemed unnecessary according to the same regulations. All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication: Not applicable.

Funding: Not applicable.

Availability of data and materials: The datasets generated and/or analyzed during the current study are not publicly available due to reasons of sensitivity and licensing requirements. For more information, contact [email protected].

Competing interests: The authors declare that they have no competing interests.

Authors' contributions: II and BD wrote the manuscript; LZ oversaw authoring the virtual cases; OK was in charge of data analysis and presentation. All authors read and approved the final manuscript.

Acknowledgment: The authors are grateful to Mr. Eugene Pozniak for his critical reading of the manuscript

Accreditation Council for Continuing Medical Education. CME Content: Definition and Examples. [Internet]. [date unknown] [cited 2023 Mar 20]. Available from: https://www.accme.org/accreditation-rules/policies/cme-content-definition-and-examples.
Josseran L, Chaperon J. Histoire de la formation médicale continue aux Etats-Unis [History of continuing medical education in the United States]. Presse Med. 2001;30:493-497.
AAFP Foundation. Chronology of continuing medical education 1947-1999. [Internet]. [date unknown] [cited 2023 Mar 20]. Available from: https://www.aafpfoundation.org/content/dam/foundation/documents/who-we-are/cfhm/factsonfile/1947-1999CMEChronology.pdf.
Cervero RM, Gaines JK. The impact of CME on physician performance and patient health outcomes: An updated synthesis of systematic reviews. J Contin Educ Health Prof. 2015;35(2):131-138.
Moore DE. How physicians learn and how to design learning experiences for them: An approach based on an interpretive review of evidence. In: Hager M, Russell S, Fletcher SW, editors. Continuing education in the health professions: Improving healthcare through lifelong learning. New York, NY: Josiah Macy, Jr. Foundation; 2007. p. 50-55.
Moore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15.
Moore DE, Chappell K, Sherman L, Vinayaga-Pavan M. A conceptual framework for planning and assessing learning in continuing education activities designed for clinicians in one profession and/or clinical teams. Med Teach. 2018;40:904-913.
Mazmanian PE, Davis DA, Galbraith R. Continuing medical education effect on clinical outcomes. Chest. 2009;135(3 Suppl):49S-55S.
Parker FW, Mazmanian PE. Commitments, learning contracts, and seminars in hospital-based CME: Change in knowledge and behavior. J Contin Educ Health Prof. 1992;12(1):49-63.
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA. 2006;296(9):1094-1102.
Ryall T, Judd BK, Gordon CJ. Simulation-based assessments in health professional education: A systematic review. J Multidiscip Healthc. 2016;9:69-82.
Berman NB, Durning SJ, Fischer MR, et al. The role for virtual patients in the future of medical education. Acad Med. 2016;91:1217-1222.
Kononowicz AA, Woodham LA, Edelbring S, et al. Virtual patient simulations in health professions education: Systematic review and meta-analysis by the Digital Health Education Collaboration. J Med Internet Res. 2019;21:e14676.
Lucero KS, Spyropoulos J, Blevins D, Warters M, Norton A, Cohen J. Virtual Patient Simulation in Continuing Education: Improving the Use of Guideline-Directed Care in Venous Thromboembolism Treatment. J Eur CME. 2020;9(1):1836865., 1-5.
Iancu I, Pozniak E, Draznin B. Virtual patient simulation platforms challenging traditional CME: Identification of gaps in knowledge in the management of Type 2 diabetes and Hyperlipidaemia. J Eur CME. 2021;10(1):1993430.
Iancu I, Zehavi L, Draznin B. Virtual Patient Simulation Offers an Objective Assessment of CME Activity by Improving Clinical Knowledge and the Levels of Competency of Healthcare Providers. J Eur CME. 2023;12(1):2166717.
Lee, J., Hammer, J. (2011). Gamification in education: What, How, Why bother? Academic exchange quarterly. 15(2):1-5.
Dichev, C., Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14 (9). https://doi.org/10.1186/s41239-017-0042-5
Jordan S. Educational input and patient outcomes: Exploring the gap. J Adv Nurs. 2000;31(2):461-471.
Sibley JC, Sackett DL, Neufeld V, Gerrard B, Rudnick KV, Fraser W. A randomized trial of continuing medical education. N Engl J Med. 1982;306(9):511-515.
Davis D, O’Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867-874.
Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66(12):762-769.
Eva KW, Regehr G. “I’ll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008;28(1):14-19.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65:S63-S67.
Edelbring S, Dastmalchi M, Hult H, Lundberg IE, Dahlgren LO. Experiencing virtual patients in clinical learning: a phenomenological study. Adv Health Sci Educ Theory Pract. 2011;16(3):331-345.
Sim JJM, Rusli KDB, Seah B, Levett-Jones T, Lau Y, Liaw SY. Virtual Simulation to Enhance Clinical Reasoning in Nursing: A Systematic Review and Meta-analysis. Clin Simul Nurs. 2022;69:26-39.
Lam EW, Lam IC, Schwartz ME, Woo D, Paull DE. Case-based virtual patient simulation impact on medical and graduate medical educational competencies and milestones: a meta-analysis. J Contemp Med Educ. 2021;11(1):6-26.
Berman N, Fall LH, Smith S, Levine DA, Maloney CG, Potts M, Siegel B, Foster-Johnson L. Integration strategies for using virtual patients in clinical clerkships. Acad Med. 2009;84(7):942-949.
Leviatan I, Oberman B, Zimlichman E, Stein GY. Associations of physicians’ prescribing experience, work hours, and workload with prescription errors. J Am Med Inform Assoc. 2021;28(6):1074-1080.
Krishnamurthy K, Selvaraj N, Gupta P, Cyriac B, Dhurairaj P, Abdullah A, Krishnapillai A, Lugova H, Haque M, Xie S, Ang ET. Benefits of gamification in medical education. Clin Anat. 2022;35(6):795-801.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Measuring Outcomes of Educational Activities with a Novel Virtual Patient Simulation-Gamification Platform

Status:

Version 1

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1