CERTAIN Program: Logic Model for System Level Planning and Evaluation of Innovative Education Delivery and Dissemination


 Background

CERTAIN (Checklist for Early Recognition and Treatment of Acute Illness and iNjury) has been shown to improve critical care process and patient outcomes in international ICUs with variable resources.
Methods

CERTAIN education program derived from this approach is designed, promoted, and implemented following the Logic model. Through the roadmap of the Logic model, we presented a dynamic, longitudinal implementation framework that had sufficient rigor yet offers flexibility to reach the need of the existing and emerging diversified medical education projects.
Results

Using the Logic model, the delivery of the CERTAIN education program is optimized to deliver relevant education content in various environments. During the COVID-19 outbreak, the implementational framework demonstrated that it could serve as an excellent template for effective response to global pandemics.
Conclusions

The Logical model is useful as a facilitation tool for planning and evaluating innovative education delivery and dissemination. The CERTAIN program provided an example for other continuous professional education projects.


Results
Using the Logic model, the delivery of the CERTAIN education program is optimized to deliver relevant education content in various environments. During the COVID-19 outbreak, the implementational framework demonstrated that it could serve as an excellent template for effective response to global pandemics.

Conclusions
The Logical model is useful as a facilitation tool for planning and evaluating innovative education CERTAIN (Checklist for Early Recognition and Treatment of Acute Illness and iNjury) is designed and developed to standardize the approach to the evaluation and treatment of acute illness and injury patients. The CERTAIN program has been shown to improve critical care processes and patient outcomes in a recent large trial in 34 ICUs with 15 countries with variable resources [1,2]. Based on the success of this trial, we designed a CERTAIN educational program to speed the dissemination of this approach to hospitals around the world. To date, 1119 inter-professional learners from 36 countries, and 234 hospitals attended 10 CERTAIN live courses (four in Rochester, MN, USA, and six in international cities from Asia and Europe) and one online course in 2020 [3]. (see course schedule le in the supplements).
In order to better develop and evaluate CERTAIN series courses, we used the Logic model, which is commonly used in public health to guide our course design, development, and delivery efforts [4][5][6]. So continuous professional education course can be better disseminated worldwide systematically and with a more signi cant impact.

Methods
The Logic model is a planning tool consisting of a matrix that provides an overview of a project's goal, activities, and expected results to improve the planning, implementation, management, monitoring, and evaluation of projects. It provides a structure to help specify the components of a project and its activities and relate them to one another. The framework is how to structure the project's main elements and highlight the linkages between them [7][8][9]. Now many large international agencies use the Logic model to guide project design. The Logic model provides the information we have developed for each component and how that information can be used to build or re ne the CERTAIN course. The components of the Logic model mainly include resources, activities, outputs, outcomes, and impact. And they represent respectively "what do I need," "what do I do," "what happens immediately?", "what are my goals?" and "what will happen." The Logic model draws a diagram illustrating the complex relations between these project elements to show how the project is designed to work for the effect of its participants. A good Logic model meets these criteria: 1) It only focuses on the project itself. 2) The outcomes are meaningful and measurable.
The result that changes in participants are logically from program implementations. Intermediate outcomes are behaviors and signi cant milestones for which the program is committed to being accountable. 3) Activities show the speci c education "dosage" and duration needed for the entire process. 4) Diverse stakeholder opinions are included in designing and developing the Logic roadmap. 5) The presentation is clear, detailed, and concise. Some Logic framework formats also depict complex in uences of the project itself [7][8][9]. In comparison, a basic Logic model should include the essential components of inputs, activities, outputs, and outcomes. The more complex versions can include a depiction of the community and organizational contexts and in uences and show the feedback loop of learning that results from evaluating program data.
The Logic model helped our program show details on measurable goals, objectives and communicate more e ciently with participants, program sites, funders. It also helped to support program implementation while expanding to other countries and offering a quick screenshot of how the project works and what will happen.
The primary objectives of the CERTAIN course are: (1) to implement the CERTAIN approach in a various hospital setting globally and evaluate the impact of the delivery on the course processes and outcomes.
(2) to create a community of faculty as change agents on the local hospital to further disseminate the CERTAIN program beyond the local hospital and community. The key outcomes of the course are related to better dissemination of the CERTAIN approach and better health care in clinical practice. To ensure the effectiveness of the CERTAIN course delivery, access to use the electronic survey support will be coupled with the uniformly available on in-person or virtual. The proposed project consists of ve variables (Inputs, Activity, Output, Outcome, Impact).
Due to the COVID-19 global pandemic, we cannot deliver the course in person because our initial program was developed from the ground up using virtual training and simulation. Therefore, it was straightforward for us to pivot the live course to a virtual course based on the theoretical model [10].

Results
Based on the Logic model as a roadmap, we designed and implemented the CERTAIN courses according to the following sections ( Figure 1).

Inputs
Firstly, we conducted a need assessment for various study sites, including the local resource, support, and faculty, before launch the education program.

(1) Resources and Program Infrastructure
We evaluated the essential education resources of leaner and their institution. Online education resources (literature links, professional society website, videos, etc.) were provided to ensure course participants would receive up to high-quality continuing medical education with minimal expenses.
(2) Technology to Support Program Management Technology support was essential for project development. We used Zoom (Zoom Video Communication, Inc) as the secure, HIPPA compatible video conferencing solution for our education program. The CERTAIN course also used several advanced tools for program management: Google Drive ( le sharing), Airtable (learner and faculty management), Articulate (Remote simulation), Trello (project management), YouTube, Blackboard and Ethos (Learning Management System), Time zone converter. By leverage those digital technologies, the CERTAIN course could be initiated and implemented from anywhere with a high-speed internet connection to learners around the world [11,12].

(3) Financial Support
Financial resources were necessary to support the program operation. The seed funding for the CERTAIN project was supported by the Mayo Clinic Education grant, following by Chest Foundation, Laerdal Foundation, WHO funding during different stages of development and dissemination. Those funding are primary to support the core program operation and coordination. We are also encouraging partner hospitals to apply for grants in local regions to support local operations.

(4) Faculty
The faculty pool was the foundation on which the CERTAIN educational program could continue developing and implementing globally. The primary faculties came from original CERTAIN investigators and further expanded to multidisciplinary team members (physicians, fellows, residents, nurses, pharmacists, respiratory therapists, etc.). In addition, international CERTAIN investigators from 15 countries joined us as local faculty when we conducted international courses in their country. More than 20 clinical fellows, research scientists, and research fellows were part of the CERTAIN team during their training and rotation over the years. Many of those fellows came from counties of CERTAIN programs were implemented. Their understanding of the local language and culture helped the CERTAIN program's local dissemination. They also helped CERTAIN modi cation based on the local practices and later shared with other countries' sites. We also offer train-the-trainer faculty development courses to prepare the faculty pool for program expansion.
With the vision to minimize preventable death, disability and expensive complications in acutely ill patients, during the CERTAIN study, we were able to recruit international centers shared the same mission to create and deliver innovative education program, and disseminate the CERTAIN approach globally, and helping physicians and nurses implement meaningful change in acute care hospitals. This culture alignment has proven to be very powerful and effective to allow the team to overcome many barriers to achieve the goal. It also fosters the CERTAIN learning community worldwide to share experiences during all phases of the learning process. We build a culture of a CERTAIN team that had the feedback from peers, educators, and technology greatly in uence learner satisfaction, and it must be harnessed to provide practical learning experiences. We are also building the train-the-trainer course and faculty playbook to re ect the course value of collaboration and teamwork. In order to design and deliver an innovative education program of a checklist-based approach, we also used design thinking and change management principles to develop standard program management to support idea generation to course delivery [13]. (1) Need Assessment We did a qualitative interview that activity as a needs assessment to help customized our education program to a speci c hospital group. By adopting qualitative methodology to be carried out by one-to-one interviews or as issue-directed discussions for getting the needs assessment [14,15]. Q-sort survey methodology is a systematic method to investigate participants' perspectives who represent different opinions on education needs by having course participants rank and sort a series of statements regarding their interests. [16,17] Participant responses were analyzed using factor analysis with ve basic steps: setting up that de nition of the domain of the particular issue, development of the statements, selection of the participants representing different perspectives, Q-sort by participants, and analysis-interpretation.
(2) Curriculum Design We chose and built the speci c curriculum across multiple specialties to match the needs of the local program. Built on the solid preliminary data and state-of-the-art implementation science, a multimodal education program consisting of asynchronous on-demand video curriculum, virtual simulation workshop, and video-enabled in-situ weekly case-based coaching and quality improvement project. The 40-week CERTAIN Programs allow the learning and implementation to occur over time in the local ICU setting.
(3) Professional Development The CERTAIN education program was created by the Mayo Clinic team based on the clinical experience of multiple medical centers. Furthermore, the local champion is a critical part committed to learning activities that are scholar-driven, manger-enable, and organization-supported. The local champion from those hospitals needs to engage key stakeholders, identify sources of funding, coordinates operation, and monitor the implementation [18]. We communicated with the local champion to lead the implementation of the course through video conference platforms to increase the scholars' participation, engagement, and social dynamics learning. All learners were encouraged to have an individual development plan working towards a continuous improvement goal. Meanwhile, local champions were expected to own the plan and keep their improvement moving forward.

(4) Scholar Activity
Beyond the main quality improvement study published, we investigated various topics during the entire life cycle of program intake, design, development, testing, training, and evaluation. The study topics covered various areas, including education research, quality improvement, patient/provider survey, and clinical research. Because we were using various techniques, that platform provides reliable data for analysis. We were also able to mentor many research trainees to develop different research projects to pursue scholarship in variable areas. After graduation from our research program, those trainees who move to other areas can further expand the future CERTAIN network.

Outputs (what happens immediately?)
The output items of the educational program had a variable consequence with the continuous deepening and promotion of the CERTAIN project. We measured the output variables of the program for many dimensions, which include patients, clinicians under the Logic framework.
(1) Learner Output As the output item of scholar professional development, the CERTAIN learner would master the skill to conduct a structural and systematic checklist approach and deliver humane and patient-centered care. We currently have 1119 learners from 36 countries, and 234 hospitals completed a live or online CERTAIN training program with a certi cate. We established a core faculty team via different development pathways through diversi ed course delivery mechanisms (live, international, remote, train-the-trainer, etc.). This mechanism would greatly increase the faculty pool for the continued international expansion in the future.
(2) CERTAIN Main Study CERTAIN was a real-time electronic decision aid that offers a systematic approach to perform an initial assessment and ongoing evidence-based management of the critically ill [19]. Since 2014, our team had successfully implemented the CERTAIN training program in a network of 36 hospitals from low-middle income countries (www.icertain.org) using web-based remote simulation and coaching. More than 900 physicians and nurses had completed the CERTAIN training program, and >5000 patients had been enrolled in the international clinical trial (ClinicaltTrials.gov NCT01973829). Recently published data had shown that CERTAIN implementation was feasible and was associated with better adherence to basic critical care processes, decreased intensive care unit (ICU) and hospital length of stays (LOS), and improved survival [1,2].

(3) Auxiliary Study
We had published study results in numerous journals and presented them in various settings (e.g., Society of Critical Care Medicine Annual Meeting, Karolinska Institute-Mayo Annual Meeting, Mayo Clinic Annual Instructional Design Educational Activities meeting). The study data set is also open for investigators for secondary analysis with a different hypothesis.
(4) Education Program: Our team created a multimodal course consisting of an asynchronous self-paced online curriculum, synchronous simulation workshop, and synchronous video-enabled remote coaching and quality improvement program. The CERTAIN remote coaching is a year-long in situ ICU remote coaching program that offers a longitudinal, weekly, and interactive virtual learning experience with Mayo Clinic critical care experts and program to facilitate and advance critical care quality improvement efforts at the unit or institutional level. The program offers a monthly core critical care curriculum, weekly case-based discussions, journal clubs, along with opportunities to develop, discuss, and collaborate on research and quality improvement projects.

Outcomes (what are my goals?)
We created the core program assessment outcomes to ensure every component can contribute data on key learning outcomes using the standard Kirkpatrick model [20,21].
(1) Short Term Outcome a. Online Quiz: We created a series of questions on the learning management system (Blackboard/ Ethos). A quiz is a set of questions that are graded to measure learners' performance and process of study. Quiz results are scored and reviewed by faculty.
b. Engagement: Beyond using Zoom, we used various asynchronous tools to promote learner engagement. In addition to the "icertain.org" website, we shared CERTAIN contents via YouTube, Twitter, message app (Viber, WhatsApp, WeChat). Those platforms have analytics dashboards to guide our engagement efforts.
c. Post Course Survey: Participants received an electronic, anonymous post-survey that collected basic demographic data and feedback on the course content and faculty using a series of questions and a 5point Likert scale during all CERTAIN programs. Survey questions were developed and re ned for clarity and content through iterative pilot testing.
(2) Long Term Outcome: a. Hospital Safety Culture: Beyond clinical outcome, the CERTAIN remote education program also has the potential to change the perception of the beliefs, attitudes, values, behavioral characteristics of participants and affect staff member attitudes and behaviors [22][23][24]. Moreover, higher culture of patient safety score has been shown to be associated with better patient outcomes [25]. We are in the process of assessing the safety culture of clinicians in different counties using the validated instrument (SAQ, Safety Attitudes Questionnaire) [22,26,27].
b. Patient improvement: Through the CERTAIN research study, compliance of best practices and patientcentered outcomes were measured by local providers. The participants were trained to harness the power of international collaboration to interact and share ideas and solutions with other local champions in the CERTAIN network. The structured, video-assisted tele-education program can effectively improve bedside compliance of evidence-based care and improve patient-centered outcomes in various intensive care units abroad [2]. However, we could not track the changes for those participants who attended only short education programs (live courses). Only those who participated in the longitudinal program (remote coaching) measured the clinical impact before and after the intervention.
Impact (What will be led to by these outcomes?)

(1) Continues Professional Education
Through CERTAIN remote coaching, we can share relevant critical care best practices with other healthcare professionals from all over the world, even during the COVID-19 global pandemic [28]. It can also improve the scholars' professional knowledge and enhance the desire and interest in clinical research. Meanwhile, the program will cultivate advanced educational ability as the output of continuous professional education, which is the foundation for better clinical work and research in the future.
(2) Healthcare Delivery It will enable clinicians to improve their knowledge and attitude toward high-quality care. Those learning will power them to process improvement during the daily practices based on KSA (Knowledge-Skills-Abilities) model. Through the study and learning during the CERTAIN program, the learner can use the knowledge gained to further advance the eld of critical care medicine in their hospital. The CERTAIN learning network is supportive for lifelong learning to transform healthcare delivery.

(3) Patient Care
Through the remote coaching of CERTAIN's structured and systematic checklist mode, the critical care best practices can be implemented more reliable in those hospitals. In addition, through the cultivation and improvement of individuals, implementing CERTAIN coaching in a critical care setting with limited resources resulted in lasting changes in structure and processes. It was associated with improvement in outcomes of critically ill patients and marked cost-savings [2].

Discussion
Based on Logic theoretical model, we have formed a sophisticated, comprehensive, e cient, and welldeveloped work ow diagram, CERTAIN education program had successfully promoted in several countries around the world during the past 6 years. Meanwhile, during the COVID-19 global pandemic, the CERTAIN team started one program for rapid critical care knowledge exchange to protect critically ill patients and staff during COVID-19 pandemics in Balkans counties. In addition, to provide concise practice guidelines endorsed by international organizations, the CERTAIN team delivered a remote video case discussion supplement with asynchronous chat via Viber to foster rapid knowledge exchange among clinicians. This model is currently being conducted in multiple worldwide sites (US, China, Bosnia, Montenegro). Tele-education interventions using social media platforms were feasible, low-cost, and effective methods to improve clinical quality, especially during the global pandemic [28].
There are several limitations from our work. The small learners' participant from a single hospital may limit the impact of the CERTAIN program. As the organizational process change will take a long time by the multidisciplinary team. Currently, the program operation and management are still labor-intensive, limiting the potential growth of the program for large scale internationally. The innovative learning management system (LMS) is in great need to further disseminate CERATIN globally. Due to the changing clinical environment, the CERTAIN course framework may need to adapt to changes. Due to the variation in resource constraints, tracking clinical impact for various centers is still a labor-intensive process. We also lack long-term follow-up data to track the learning and education impact beyond survey and clinical outcome in a limited observation period.

Conclusions
To disseminate the CERTAIN approach worldwide using innovative education program is feasible to be customized for local institutions and various settings. Using the Logic model to design and evaluate impact provides valuable feedback to improve the program. It helps to de ne the structure of data elements and set relationships between them. Our program response to the COVID-19 outbreak demonstrated that the multidisciplinary team's agility is adaptable to the ever-changing healthcare delivery landscape and dissemination. CERTAIN approach with Logic model provides foundational structural and systematic practices to maximize knowledge transfer. It could also serve as a template for many other global continuous professional development education programs in different countries/cultures, including pandemic time.

Consent for publication
All authors consent for publication.
Availability of data and materials