Between June 2012 and December 2017, 55 study centers from 20 countries were recruited to participate; of those 55, 46 received IRB approval and 44 completed study setup. The first center entered the CERTAIN QI study after obtaining local IRB approval in June 2012 and collected the first patient baseline data on November 31, 2013. Of the 44 centers that completed study setup, 38 (86%) completed baseline data collection (stage 1), 38 (86%) completed remote simulation training (stage 2), 35 (80%) completed local implementation (stage 3), and 34 (77%) completed post-implementation data collection and maintenance of processes of care (stage 4) and completed the entire study (Fig. 3).
Measure Phase
Among the centers, the time needed to complete each stage and to complete the entire study varied considerably (Table 3).
Table 3. Time to Complete Each Stage |
Study stage | No. of Centers | Time, median (IQR), wk |
Setup period | 44 | 5.8 (1.6–18.5) |
Baseline data collection (stage 1) | 38 | 16.1 (10.3–29.4) |
Remote simulation training (stage 2) | 38 | 7.1 (3.8–16.1) |
Local implementation (stage 3) | 35 | 15.0 (8.6–26.6) |
Post-implementation data collection and maintenance of processes of care (stage 4) | 34 | 30.2 (19.3–45.0) |
Completion of whole study (stages 1–4) | 34 | 96.1 (54.0-139.3) |
Abbreviation: IQR, interquartile range. |
Analyze and Improve Phases
The most commonly identified challenges during implementation of the CERTAIN QI project included the following 6 domains: 1) leadership support and team building, 2) communication, 3) culture environment, 4) language barriers, 5) infrastructure and technology, and 6) sustainability (Table 1). To overcome these challenges, different strategies were deployed.
Table 1. Levels of Difficulty in 6 Domains for implementation of CERTAIN in 20 Countries |
Study Stage | Leadership Support and Team Building | Communication | Work Environment Culture | Language Barriers | Infrastructure and Technology | Sustainability |
Setup period | Diff | Diff | Diff | Mod | Diff | Easy |
Stage 1a | Diff | Mod | Easy | Mod | Diff | Easy |
Stage 2b | Diff | Mod | Mod | Diff | Diff | Easy |
Stage 3c | Mod | Diff | Diff | Diff | Mod | Mod |
Stage 4d | Mod | Diff | Diff | Mod | Mod | Diff |
Abbreviations: CERTAIN, Checklist for Early Recognition and Treatment of Acute Illness and Injury; Diff, difficult; Mod, moderate. a Baseline data collection. b Remote simulation training. c Local implementation. d Post-implementation data collection and maintenance of process of care. |
Leadership Support and Team Building
The engagement and credentials of the principal investigators at each institution substantially affected the success of CERTAIN implementation. If a principal investigator was a departmental chair or an ICU leader, the project moved forward smoothly. Without the support from leadership, some centers encountered resistance to implementation of CERTAIN within their local practice. This led to delays in reaching study milestones and even withdrawal from the study. With this in mind, we encouraged the investigators to engage the leadership at their institutions as soon as they were considering participation.
We also realized that personnel shortages impeded CERTAIN implementation. We communicated with the local champions in advance and made suggestions about involving other potential collaborators, including residents and medical students to assist with data collection after they received adequate training of the CERTAIN study protocol.
As the study progressed, we noticed that we could not have trainees come back to complete remote simulation training because of scheduling difficulties across different time zones. Those trainees were not able to be certified to use CERTAIN and train other users. Therefore, we started scheduling training at locally convenient times for busy clinicians who later became CERTAIN local champions.
Communication
CERTAIN study centers were located in 20 countries across 5 continents, and communication was challenging. We built an active online user community with various communication tools, such as a CERTAIN website, monthly newsletters, webinars, social media, email reminders, and CANVAS (Instructure, Inc), a web-based learning management system. Occasionally, we conducted site visits during travel in the region because many study team members lived in countries where CERTAIN study centers were located. Interestingly, we found that different countries had preferred communication tools. For example, CERTAIN champions in North America and Europe responded timely by email, while champions in Africa and Asia responded quickly through social media.
To guarantee the quality of data during data collection in the baseline stage and the final stage, we provided an SOP template and online list of frequently asked questions (FAQs). We responded to address questions and technique issues quickly through the communication tools mentioned above, and we updated the FAQs periodically. Web conferencing was essential for timely communication.
Culture Environment
Some centers and their principal investigators encountered resistance to change. This directly affected study implementation, including the earliest stage—receiving IRB approval. The lack of support from the clinical team interfered with implementation and sometimes completely prevented it. We used data from a previous simulated study and clinical outcome from the pilot center to encourage IRBs that the study was important and feasible.
We also used the ADKAR (Awareness, Desire, Knowledge, Ability, and Reinforcement) change management model [7]. We developed online training materials in the web-based learning management system and sent hard copies of the CERTAIN booklet and an online CERTAIN tool. We sent friendly reminders and followed up with local champions to obtain feedback routinely. During the local implementation phase, local champions were all CERTAIN certified. Slowly the culture changed, and the study reached milestones.
Infrastructure and Technology
Lack of internet support and computers or alternative electronic devices at CERTAIN study centers was another large hurdle during each stage. The orginal CERTAIN module was a web-based decision support tool displaying relevant clinical information incorporated with knowledge about evidence-based best clinical practice [1]. If centers did not have access to computers and reliable internet service, we provided hard copies of the checklist and other CERTAIN material. The lack of reliable internet service delayed training at some centers. Before scheduling training sessions, we communicated with centers to make sure that the internet service was stable and unhindered by technologic problems. Sometimes local champions had to use their personal internet data plans to conduct video conferences. For 1 center, we subsidized the local principal investigator’s cell phone internet data plan so that team members could use the checklist on their mobile phones. We also encouraged local centers to seek funding and supported them with applications to enhance the study. Protocols were translated into local languages to help facilitate the preparation and submission of local research grant applications. Several centers received funding from foundations, such as the CHEST Foundation and the Laerdal Foundation, and research grants were awarded by local governments (e.g., Sichuan China, and Tianjin China).
We responded to local requests to make the CERTAIN checklist usable and accessible in different formats, including hard copy versions and portable document formats (PDFs) that could be printed, enlarged, and laminated for use with erasable markers. Later we also developed web-based CERTAIN online modules that were compatible with mobile devices to meet the users’ preferences in different centers.
A major concern was having secure data storage that would be compliant with HIPPA. CERTAIN patient data were collected and managed with Research Electronic Data Capture (REDCap) hosted at Mayo Clinic in Rochester, Minnesota [8].
Language Barriers
Local champions and team members spoke many languages besides English. They used the local languages as needed to conduct train-the-trainer sessions at the centers. The CERTAIN checklist and content were translated into many languages (Arabic, Chinese, Spanish, Mongolian, Polish, Serbian/Croatian/Bosnian, and Turkish) in PDF files or hard copies to facilitate local adoption.
Sustainability
Regular communication with the local champions was necessary to keep the study on track. We encouraged ongoing involvement in the multicenter collaborative groups as trainers, speakers, and participants in conferences.
Control Phase
Although improvement was ongoing, the CERTAIN project transitioned to the control phase on May 15, 2015, when we implemented the above changes for study centers using the DMAIC framework. We divided the study centers into 2 groups: Group 1 (non-control phase) included centers that participated in the CERTAIN project before May 15, 2015, and group 2 (control phase) included centers that enrolled on or after May 15, 2015. Compared with group 1, group 2 had smaller variation and a significant decrease in the time for study setup, baseline data collection, remote simulation training, local implementation, post-implementation data collection, and the whole study (Table 2). During the control phase, the time needed to complete each stage for group 2 was close to the proposed study time.
Table 2. Time to Complete Each Stage |
Study Stage | Group 1 Centersa (N = 19) | Group 2 Centersb (N = 25) | P Value |
No. | Median (IQR), wk. | No. | Median (IQR), wk. |
Set-up period | 19 | 11.7 (4.9–39.9) | 25 | 2.3 (0.7–10.5) | .006 |
Stage 1c | 17 | 20.3 (15.1–35.7) | 21 | 12.9 (4.7–27.6) | .02 |
Stage 2d | 17 | 17.6 (11.3–22.2) | 21 | 4.6 (2.9–6.6) | < .001 |
Stage 3e | 14 | 24.8 (19.8–71.8) | 21 | 9.6 (6.4–14.4) | < .001 |
Stage 4f | 14 | 45.2 (28.3–62.5) | 20 | 23.9 (18.4–33.4) | .004 |
Completion of whole study (stages 1–4) | 14 | 162.3 (116.3-196.6) | 20 | 67.2 (40.5–90.3) | < .001 |
Abbreviation: IQR, interquartile range. a Entered the study before May 15, 2015. b Entered the study on or after May 15, 2015. c Baseline data collection. d Remote simulation training. e Local implementation. f Post-implementation data collection and maintenance of process of care. |