Implementation of a Global Quality Improvement Project in 20 Countries: A Case Study

Background: The Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) study is a before-and-after clinical quality improvement study involving 20 countries. Our aim was to describe the challenges and solutions for study implementation across various intensive care units. Methods: After local institutional review board approval, each study center had a setup period before patient data were collected. Each center was required to accomplish 4 stages: baseline data collection, remote simulation training, local implementation, and postimplementation data collection and maintenance of process of care. We measured the time required to complete each study stage and used the Dene, Measure, Analyze, Improve, and Control (DMAIC) framework to identify the challenges encountered during each stage and guide improvement strategies to facilitate implementation of CERTAIN. Results: Between June 2012 and December 2017, 55 centers from 20 countries were recruited to participate. The time to complete each stage at different centers varied signicantly. The most common domains in which challenges occurred were 1) leadership support and team building, 2) communication, 3) work environment culture, 4) language barriers, 5) infrastructure and technology, and 6) sustainability. Obstacles in these 6 areas delayed reaching major milestones of the study. Different strategies were deployed to overcome these barriers: engaging leadership for support, using native-speaking clinicians during training sessions, using various communication platforms, providing personal computers to local hospitals, making in-person site visits, offering exible training time, and supporting local centers’ grant applications. Centers that enrolled during the control phase, compared with those that enrolled earlier, had signicantly shorter times and less variation in times for each study stage and for the total study time. Conclusions: By using the DMAIC method, we identied the predominant 6 domains in which challenges occurred during implementation of the global CERTAIN study and developed targeted strategies to overcome the challenges and shorten the duration of the study. The DMAIC method would be useful in improving the eciency of other large-scale multinational quality improvement projects.


Background
The Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) study was a multinational quality improvement (QI) clinical study, designed and developed to standardize the approach for the assessment and management of critically ill patients in acute care hospitals [1]. The primary goal of this clinical QI study was to implement the CERTAIN clinical decision support tool in diverse acute care environments around the world to minimize preventable death, disability, and expensive complications in the sickest patients [1]. CERTAIN offers an evidence-based checklist for intensive care unit (ICU) rounds and an admissions checklist in electronic formats (for mobile or desktop devices) or non-electronic formats. In addition, CERTAIN content provides point-of-care information to support enhanced clinical decision making for critically ill patients with common acute syndromes and for related procedures and medications [2].
Owing to the large scale of implementation in widely diverse hospital settings, during the early period of the study our team observed delays due to various obstacles and challenges. The complexity of implementing the CERTAIN QI project necessitated a structured approach. To better understand the process issues, we used the De ne, Measure, Analyze, Improve, and Control (DMAIC) method, a datadriven, customer-focused, structured, problem-solving framework that has been adopted from other industries and widely applied in health care settings [3]. McJoynt and colleagues [4] used the DMAIC framework to improve the e cacy and e ciency of cancer protocol development and noted that both the internally authored protocol and the externally authored protocols were signi cantly improved. With the use of the DMAIC method, we developed mechanisms to improve implementation efforts, e ciency, and success of the interventions within our budget constraints during implementation of CERTAIN in 20 countries. Through the engagement of local investigators and their teams, we gained valuable experience and useful insights into the implementation of QI projects in low-and middle-income countries. The purpose of this article is to describe our experience with using the DMAIC framework to address the obstacles encountered during implementation of the CERTAIN QI project in 20 countries.

Methods
Brief Review of the CERTAIN Study CERTAIN was implemented with a modi ed stepped-wedge design ( Figure 1) [1]. When a center received institutional review board (IRB) approval, the center was considered to be participating in the study. The period from IRB approval to recruitment and collection of data from the rst patient was de ned as the study setup period. Waiver of patient informed consent was approved by local IRBs. After study setup, each center was required to accomplish 4 sequential stages: 1. Baseline data collection: During this stage, each center collected data for 3 months or data from at least 50 patients per ICU.
2. Remote simulation training [5]: When the baseline data had been collected, the center participated in remote simulation training, which included baseline training without integration of the CERTAIN tool and an advanced training session with integration of the CERTAIN tool.
3. Local implementation: After remote simulation training sessions were completed, the center started local implementation of CERTAIN. During local implementation, all CERTAIN centers were required to attain a compliance rate of 80% for the ICU rounds and admissions checklists for all eligible patients [1]. The center then entered the nal stage.
4. Postimplementation data collection and maintenance of processes of care: During the nal stage, each study center collected data for 6 months or data from at least 100 patients per ICU. When data were collected for the last patient, the center was considered to have completed the nal stage of CERTAIN.
De ne Phase: Delays in Completion of Each Implementation Stage Mayo Clinic was the main study center for coordinating the training and implementation efforts and for data collection for all CERTAIN centers. The CERTAIN team at Mayo Clinic was composed of experts in conducting observational research and QI studies. During the early period of the CERTAIN project, the time to complete each stage of the study was much longer than the proposed time in the study protocol ( Figure 2). The CERTAIN team aimed to shorten the time to complete the study for all new centers as the stepped-wedge study design continued. To encourage e cient implementation and to minimize prolonged study completion time, we applied the DMAIC framework to streamline the study stages.
Measure Phase: Duration of Each Implementation Stage With the modi ed stepped-wedge design, each center entered the study stages at different times. When each center entered the study, we used a spreadsheet (Excel® Microsoft Corp) and an online project management tool (Trello® Atlassian) to track the start and nish time of each major study stage. We monitored these time points and measured the time that each study center spent in each stage of the study.

Analyze Phase: Barriers for Delay
We reviewed and analyzed the data weekly to determine the future strategic efforts for implementation.
The global CERTAIN team met monthly through a remote conferencing service (Zoom Video Communications, Inc.) that included mechanisms for complying with the Health Insurance Portability and Accountability Act (HIPPA) and offered remote conferencing services with cloud computing [6]. We used process mapping, phone calls, emails, in-person meetings, and social media communication to encourage timeliness in completion of each stage of the study. We applied root cause analysis to highlight the barriers that might impede local implementation of CERTAIN. After discussions with local investigators, we identi ed implementation barriers in 6 domains: 1) leadership support and team building, 2) communication, 3) work environment culture, 4) language barriers, 5) infrastructure and technology, and 6) sustainability (Table 1).

Improve Phase: Deployment of Improvement Strategies
To improve the implementation process at each center, we modi ed our approaches according to each center's experience. Strategies included engaging site leadership for implementation support, early scheduling of training at a locally convenient time, and supporting and encouraging centers to apply for regional funding to keep the study viable and meet major study milestones. We clari ed the timeline and communicated with local leaders and champions. We set a goal to improve e ciency and applied the Plan-Do-Study-Act (PDSA) cycle whenever a new center participated. We also facilitated knowledge sharing between centers to disseminate knowledge about approaches to improve the effectiveness of implementation (Table 1). Those fast iteration improvement strategies were also shared with all investigators globally during the monthly electronic conferences.

Results
Between

Measure Phase
Among the centers, the time needed to complete each stage and to complete the entire study varied considerably (Table 3). Abbreviation: IQR, interquartile range.

Analyze and Improve Phases
The most commonly identi ed challenges during implementation of the CERTAIN QI project included the following 6 domains: 1) leadership support and team building, 2) communication, 3) culture environment, 4) language barriers, 5) infrastructure and technology, and 6) sustainability (Table 1). To overcome these challenges, different strategies were deployed. Leadership Support and Team Building The engagement and credentials of the principal investigators at each institution substantially affected the success of CERTAIN implementation. If a principal investigator was a departmental chair or an ICU leader, the project moved forward smoothly. Without the support from leadership, some centers encountered resistance to implementation of CERTAIN within their local practice. This led to delays in reaching study milestones and even withdrawal from the study. With this in mind, we encouraged the investigators to engage the leadership at their institutions as soon as they were considering participation.
We also realized that personnel shortages impeded CERTAIN implementation. We communicated with the local champions in advance and made suggestions about involving other potential collaborators, including residents and medical students to assist with data collection after they received adequate training of the CERTAIN study protocol.
As the study progressed, we noticed that we could not have trainees come back to complete remote simulation training because of scheduling di culties across different time zones. Those trainees were not able to be certi ed to use CERTAIN and train other users. Therefore, we started scheduling training at locally convenient times for busy clinicians who later became CERTAIN local champions.
Communication CERTAIN study centers were located in 20 countries across 5 continents, and communication was challenging. We built an active online user community with various communication tools, such as a CERTAIN website, monthly newsletters, webinars, social media, email reminders, and CANVAS (Instructure, Inc), a web-based learning management system. Occasionally, we conducted site visits during travel in the region because many study team members lived in countries where CERTAIN study centers were located. Interestingly, we found that different countries had preferred communication tools. For example, CERTAIN champions in North America and Europe responded timely by email, while champions in Africa and Asia responded quickly through social media.
To guarantee the quality of data during data collection in the baseline stage and the nal stage, we provided an SOP template and online list of frequently asked questions (FAQs). We responded to address questions and technique issues quickly through the communication tools mentioned above, and we updated the FAQs periodically. Web conferencing was essential for timely communication.

Culture Environment
Some centers and their principal investigators encountered resistance to change. This directly affected study implementation, including the earliest stage-receiving IRB approval. The lack of support from the clinical team interfered with implementation and sometimes completely prevented it. We used data from a previous simulated study and clinical outcome from the pilot center to encourage IRBs that the study was important and feasible.
We also used the ADKAR (Awareness, Desire, Knowledge, Ability, and Reinforcement) change management model [7]. We developed online training materials in the web-based learning management system and sent hard copies of the CERTAIN booklet and an online CERTAIN tool. We sent friendly reminders and followed up with local champions to obtain feedback routinely. During the local implementation phase, local champions were all CERTAIN certi ed. Slowly the culture changed, and the study reached milestones.

Infrastructure and Technology
Lack of internet support and computers or alternative electronic devices at CERTAIN study centers was another large hurdle during each stage. The orginal CERTAIN module was a web-based decision support tool displaying relevant clinical information incorporated with knowledge about evidence-based best clinical practice [1]. If centers did not have access to computers and reliable internet service, we provided hard copies of the checklist and other CERTAIN material. The lack of reliable internet service delayed training at some centers. Before scheduling training sessions, we communicated with centers to make sure that the internet service was stable and unhindered by technologic problems. Sometimes local champions had to use their personal internet data plans to conduct video conferences. For 1 center, we subsidized the local principal investigator's cell phone internet data plan so that team members could use the checklist on their mobile phones. We also encouraged local centers to seek funding and supported them with applications to enhance the study. Protocols were translated into local languages to help facilitate the preparation and submission of local research grant applications. Several centers received funding from foundations, such as the CHEST Foundation and the Laerdal Foundation, and research grants were awarded by local governments (e.g., Sichuan China, and Tianjin China).
We responded to local requests to make the CERTAIN checklist usable and accessible in different formats, including hard copy versions and portable document formats (PDFs) that could be printed, enlarged, and laminated for use with erasable markers. Later we also developed web-based CERTAIN online modules that were compatible with mobile devices to meet the users' preferences in different centers.
A major concern was having secure data storage that would be compliant with HIPPA. CERTAIN patient data were collected and managed with Research Electronic Data Capture (REDCap) hosted at Mayo Clinic in Rochester, Minnesota [8].

Language Barriers
Local champions and team members spoke many languages besides English. They used the local languages as needed to conduct train-the-trainer sessions at the centers. The CERTAIN checklist and content were translated into many languages (Arabic, Chinese, Spanish, Mongolian, Polish, Serbian/Croatian/Bosnian, and Turkish) in PDF les or hard copies to facilitate local adoption.

Sustainability
Regular communication with the local champions was necessary to keep the study on track. We encouraged ongoing involvement in the multicenter collaborative groups as trainers, speakers, and participants in conferences.

Control Phase
Although improvement was ongoing, the CERTAIN project transitioned to the control phase on May 15, 2015, when we implemented the above changes for study centers using the DMAIC framework. We divided the study centers into 2 groups: Group 1 (non-control phase) included centers that participated in the CERTAIN project before May 15, 2015, and group 2 (control phase) included centers that enrolled on or after May 15, 2015. Compared with group 1, group 2 had smaller variation and a signi cant decrease in the time for study setup, baseline data collection, remote simulation training, local implementation, post-implementation data collection, and the whole study (Table 2). During the control phase, the time needed to complete each stage for group 2 was close to the proposed study time.

Discussion
Global collaboration expedites the completion of clinical trials [9]. With modern technology, national and intercontinental collaboration is feasible, but global collaboration is still challenging, especially with the implementation of large-scale, international clinical QI studies. Identi cation of the causes and types of challenges is crucial. Dixon-Woods and colleagues [10] identi ed 10 key challenges in improving quality in health care; the top 3 are convincing people that there is a problem that is relevant to them, convincing them that the chosen solution is the right one, and optimizing data collection and monitoring systems.
CERTAIN is a modi ed stepped-wedge cluster implementation design. The challenges were encountered across various ICU settings in 20 countries. We faced not only the challenges outlined by Dixon-Woods and colleagues but also new challenges that are not encountered in traditional QI projects, including communication di culties, language barriers, poor local funding support, and infrastructural insu ciencies. These challenges or barriers can increase the risk of failure for projects. Studies have shown that even with experienced project managers and structural project management methods, many projects fail [11]. For organizations not involved in health care, the failure rate for change projects is 35-70% [12,13]. Within health care systems, some studies have reported that QI projects can lead to decreases in mortality, morbidity, and health care costs. An evidenced-based intervention reduced rates of catheter-related bloodstream infection by up to 65% [14]. Haynes and colleagues [15] implemented a surgical checklist and reduced the death rate from 1.5-0.8%. However, many health improvement initiatives fail. For example, the World Health Organization has advocated the use of a surgical checklist, but for many hospitals introducing a preoperative checklist, the results have been mixed [16].
Within the management literature, successful implementation of the change initiative or process improvement is a heated topic, but this topic is seldom discussed in health care process improvement, and only a few studies have explored this topic. Kash and colleagues [17] identi ed 10 speci c success factors in the implementation of change initiatives after interviews were conducted with 61 health care leaders in 2 large systems. The top 3 success factors were culture and values, business processes, and people engagement [17]. Another study focused on 167 front-line leaders from 4 community hospitals to explore why hospital change efforts fail and to identify 10 primary barriers to successful hospital change, such as poor implementation, short time lines, lack of support for the project, weak leadership, suspicion toward upper management, unrealistic plans, and communication problems [18]. Pronovost and colleagues [19] recommended 6 measures for approaching challenges in health care improvement: 1) welcome all colleagues to work toward a clearly de ned goal; 2) acknowledge all notions of loss; 3) explain clearly why change is necessary; 4) communicate the bene ts at an individual level; 5) understand others' perspectives without judging; and 6) be aware of pressure at the organizational level.
Health care is one of the largest industries in the United States [11]. Dilts' work [20] explored how clinical researchers could reduce the lengthy process involved in launching human trials. He emphasized that clinical research must transform itself now to prevent a decline similar to that seen in the American automobile industry. Johnson and colleagues [21] used metrics to improve time lines and found signi cant variance among academic medical centers, compared with nonacademic centers, with delays in IRB approval of protocols and contract execution, which resulted in delays in enrolling patients and meeting enrollment targets. A few studies have shown improvements in e cacy and e ciency in conducting pharmaceutical clinical trials without compromising patient safety and the quality of clinical research [4,22].
The CERTAIN QI study involved 34 study centers in 20 countries. The lessons we learned from the centers enrolled at the beginning of CERTAIN implementation were applied to the study centers that enrolled later.
We found that centers in group 2 (the centers enrolled during the control phase) completed each stage faster than centers in group 1 with a shorter study time and smaller variation in study time. The most signi cant improvement in time e ciency was seen in the remote simulation training. The training time decreased from 17.6 weeks (group 1) to 4.6 weeks (group 2), which was closer to the proposed time of 2 to 4 weeks outlined in the study design. During the training stage, owing to more direct interactions with the local centers, the coordinating study site could control and assist with more factors.
Communication tools are important, and one of the best strategies to keep the study on the right track was regular communication with local sites and regional liaison leaders at each study stage. As seen in health care service or other elds, conventional approaches to deliver service cannot handle the current demands [23], but with the improvements in technology, large global projects are now feasible. During the CERTAIN implementation, we adapted the study to the local communication preferences across the global CERTAIN network (Fig. 4). We sent many reminders and noti cations by email and received quick responses from the US investigators, but we learned that the use of social media was more e cient and effective to reach investigators outside the United States for progress reports. Through social media, we received prompt replies from investigators in countries such as China, Pakistan, Balkan countries, and Ireland and in countries in Africa and South Asia. To receive progress updates from investigators in a few countries, we had to make phone calls. We used video conference software to reach individual centers and to conduct monthly follow up with global principal investigators.
Our team included experts in clinical research, simulation training, QI, content management, and technologic support. We also had a large critical care team who were the experts and licensed intensivists at local centers. They ensured that the content was adopted at their local centers appropriately. Our core team members met weekly to resolve issues related to implementation.
Implementation of the CERTAIN study had limitations. We used a train-the-trainer strategy, which helped to facilitate local implementation, but as we focused on engaging leadership we could not reach all users for individual feedback. Compared with other global studies and the industry standard, our study was supported with minimal funding, which may have increased our challenges and our ability to tackle them.
The scope of international study makes it even more challenging to support global collaboration [24].
Despite these limitations, we have demonstrated the feasibility to implement a large-scale, clinical QI project with the use of effective teamwork and technology. These lessons may provide guidance for other investigators considering future research in conducting and implementing multinational QI projects and other studies.

Conclusions
During the early phase of study, the time needed to complete the stages of this multinational QI study varied signi cantly. By using the DMAIC method, we identi ed the predominant 6 domains in which the challenges occurred: 1) leadership support and team building, 2) communication, 3) work environment culture, 4) language barriers, 5) infrastructure and technology, and 6) sustainability. We developed targeted strategies to overcome the challenges and measured the outcome. During the control phase of the study, the variation improved signi cantly and the length of time shortened signi cantly. The DMAIC method would be useful in improving the e ciency of other large-scale, multinational clinical QI projects.

Declarations Ethics Approval and Consent to Participate
This study was approved by Mayo Clinic Institution Review Board. Waiver of patient informed consent was approved by Mayo Clinic IRB and local study site IRBs.

Consent for Publication
All study sites and their investigators agreed to used their data for this publication. This study was approved by Mayo Clinic Institution Review Board and local study site IRBs.

Availability of Data and Material
The datasets used in the current manuscript are available from the corresponding author on reasonable request.

Funding
The HHC and LY analyzed the data. HHC and YD interpreted the data and wrote the draft of manuscript. AKB revised the manuscript thoroughly for a few rounds and gave feedback on intellectual content. LF provided computer and information technology support. MV, RK, OG, RS, SG and MH all gave feedback on the manuscript. All authors reviewed and approved the nal manuscript. OG, MV and YD applied and received funding to support this study.

Competing Interest
Mayo Clinic, OG, and RK have a potential nancial con ict of interest related to CERTAIN software, which has been licensed to Ambient Clinical Analytics, Inc. AB has potential nancial con ict of interest due to spousal connections with Ambient Clinical Analytics, Inc.    Global Centers in the Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN) study.