The German Quality Network Sepsis: Evaluation of a Quality Collaborative on Decreasing Sepsis-Related Mortality in a Quasi-Experimental Difference-in-Differences Design

Sepsis is one of the leading causes of preventable deaths in hospitals. This study presents the evaluation of a quality collaborative network, which aimed to decrease sepsis-related hospital mortality. The German Quality Network Sepsis (GQNS) offers quality reporting based on claims data, peer reviews, and support for establishing continuous quality improvement and staff education to participating hospitals. The primary outcome of the evaluation was all-cause risk-adjusted hospital mortality among cases with sepsis per hospital. Sepsis was identied by ICD-10 codes in claims data for sepsis with organ dysfunction or septic shock according to the sepsis-1 denition. To evaluate the effect of the GQNS, the change in outcomes from a retrospective baseline (January 2014 – March 2016) to the intervention phase (April 2016 – June 2018) was compared between the hospitals in the GQNS and the national German diagnosis-related-groups statistics. Tests were conducted by interrupted time-series analyses using hierarchical models. Implementation processes and barriers were assessed by a survey of local leaders of quality improvement teams in autumn 2018.

Interrupted time-series analyses did not show effects on course or level of risk-adjusted mortality of cases with sepsis compared to the national DRG-statistics after the beginning of the intervention period (p=0.632, and p=0.512, respectively). There were also no signi cant effects in the subgroups of patients with septic shock or ventilation >24h or subgroups of hospitals. Surveys among 48 local quality improvement leaders revealed that most hospitals did not succeed in implementing a continuous quality improvement program or relevant measures to improve early recognition and treatment of sepsis. Barriers perceived most commonly were lack of time (77.6%), staff shortage (59.2%), and lack of participation of relevant departments (38.8%).

Conclusions
As long as hospital-wide sepsis quality improvement efforts will not become a high priority for the hospital leadership by assuring adequate resources and involvement of all pertinent stakeholders, voluntary initiatives to improve the quality of sepsis care will remain prone to failure.

Background
Sepsis is the leading cause of death due to infectious diseases [1] and might also be the leading cause of preventable deaths in hospitals [2]. Timely recognition and adequate anti-infective treatment have been shown to decrease mortality, but awareness of sepsis is often low in everyday clinical practice [2][3][4][5][6][7][8]. A recent meta-analysis showed that performance improvement programs substantially improved implementation of sepsis guidelines including early adequate antimicrobial treatment -and decreased odds of death [9]. Such quality initiatives typically use a multifaceted approach by assessing and reporting quality, staff education, and implementing changes in care processes [9]. Prospective inclusion of patients with sepsis and documentation of clinical data for quality indicators put a high workload on participating hospitals, which can cause poor reporting or even the drop-out of hospitals from quality improvement projects [10,11].
Using claims data for performance measurement has the advantage of covering all ICD-coded cases with data readily available and needing minimal time and costs [12]. This approach is extensively used within quality initiatives in the USA [13]. It has also achieved the rst promising results in Germany, where a large quality initiative combines benchmarks of quality indicators based on administrative data with peer reviews [14].Therefore, the German Quality Network Sepsis (GQNS) was founded as a quality collaborative to support participating hospitals to improve sepsis care by offering quality reports based on claims data, peer reviews, and support to implement a continuous quality management and regular staff education. The participation in the GQNS was voluntary and the full responsibility for implementation of quality improvement measures was on the side of the participating hospitals. This study aims to evaluate the effect of hospitals' participation in the GQNS on mortality among patients with sepsis.

Context
The GQNS was founded in February 2016. The start-up period of the GQNS was funded by grants from the German Federal Ministry of Education and Research (BMBF) and ran from August 2015 to July 2018. The funded start-up phase and its scienti c evaluation used the acronym ICOSMOS (quality Improvement in infection COntrol and Sepsis management in MOdel regionS). The study was approved by the ethical review board of the Jena University Hospital (IRB protocol 4536-08/15). The necessity of informed consent by patients was waived since only pseudonymized claims data were used. Details on the concept and conduction of the GQNS, as well as the planned evaluation, are given in the study protocol (Supplemental Digital Content 1) [15]. Passages cited from the study protocol are not individually marked in the manuscript. The study description follows the Standards for QUality Improvement Reporting

Participating hospitals
Eligible for participation in the GQNS were acute care hospitals with at least one adult intensive care unit.
Invitation letters were sent to management boards of hospitals that were participating in former or ongoing sepsis-related quality initiatives or research networks, to regional and national hospital groups, and all German university hospitals. Hospitals could join the GQNS at the time of its foundation or any later time.

Project organization
The GQNS was coordinated by the central study coordinating bureau at the Jena University Hospital.
Claims data were collected and processed to generate quality reports by medical information technology service provider (3M Health Information Systems). Participating hospitals of the GQNS named a local leader of the quality improvement process. The quality improvement leaders were encouraged to establish interdisciplinary and interprofessional quality improvement teams right from the beginning of the participation in the GQNS. The formation of these teams was not mandatory and selection of the members was at the discreation of the quality improvement leader. It was suggested by the study coordinating bureau to include at least intensive care departments, the emergency department, quality management department, and medical and surgical departments responsible for inpatient treatments of adjult patients. Major decisions were made in the general assembly of representatives of all participating hospitals. This general assembly was formed by the local quality improvement leaders and met once a year in autumn. A steering committee was elected among the delegates of the general assembly to supervise the work of the coordinating bureau. Meetings of the steering committee and the study coordinating bureau were conducted by phone or webconference every few months.

Interventions
The core interventions for hospitals in the GQNS are: a) reporting and publication of quality indicators; b) case analyses within the participating hospitals; c) peer reviews for hospitals, which are outliers in the quality reports; and d) hospital-wide staff education in participating hospitals. Peer review is a process by which health care providers evaluate each others' performance [17]. The only mandatory intervention was the reporting, benchmarking, and publication of quality indicators. The study coordination bureau provided information and support regarding the conduction of case analyses and staff education and coordinated peer reviews. The full responsibility for implementation was on the side of the participating hospitals, and the participation in peer reviews was voluntary.

Reporting and publication of quality indicators
Data for assessment of quality indicators were provided by diagnosis-related group (DRG) data of each participating hospital, which were sent to the information technology service provider. These data can be exported easily in a standardized format from the hospitals' patient data management system. The service provider supplied the quality reporting quarterly to each hospital beginning in April 2016. Cases with sepsis were identi ed based on speci c ICD-10-codes for sepsis with organ dysfunction or septic shock according to sepsis-1 de nitions (R65.1: sepsis with organ dysfunction, R57.2: septic shock) [18], that were still in use in the German ICD-10 until the end of 2019. Quality reports contained incidence and risk-adjusted mortality for cases with sepsis and the subgroups of patients with septic shock, sepsis and mechanical ventilation of more than 24 hours, admission to the hospital via a surgical, and admission to the hosptital via a medical department. Hospitals' own results could be compared to other participating hospitals, subgroups of participating hospitals (primary, secondary, tertiary care, and university hospitals), the overall average in the GQNS, as well as to the average among all German hospitals. Mortality was risk-adjusted by a validated complex model developed for the GQNS, which was based on German national DRG-statistics [19]. De nitions of variables are presented in Supplemental Digital Content 2. Quality reporting also included case lists presenting predicted and observed mortality for each sepsis case sent to the hospitals, which provided the basis for case analysis and peer reviews. The study coordinating bureau provided hospitals with instructions on how to use the quality reports, and how to conduct case analyses. This was done during annual meetings and by providing educational material on the website of the GQNS. Hospitals within the GQNS consented to publish their hospitals' risk-adjusted mortality indicators compared to the average of the German national DRG-statistics after two years of participation on their own website. The rst publication of the main quality indicators occurred in Summer of 2018. Since hospitals were ought to publish their risk-adjusted mortality after two years of participation, only 11 hospitals, which had signed their contract for participation in 2015, did publish their quality indicators.

Case analyses and peer reviews
Based on the provided case lists, expired patients with sepsis with a predicted low risk of in-hospital mortality were identi ed and used to analyze and discuss possible problems in the quality of care in interdisciplinary case conferences within the individual participating hospitals [20]. The same method was used to select cases for analysis by external peers. An external peer review was suggested by the central study coordinating bureau to hospitals with the highest risk-adjusted mortality among patients with sepsis. Peers were physicians and nurses, who were recruited among the participating hospitals and had a special quali cation to conduct peer reviews. A team of at least four peers visited the respective hospital, conducted analyses of up to ten selected charts of patients with sepsis, and discussed improvement strategies with local clinicians. Peer reviews were voluntary and hospitals could refuse to take part. Six external peer-reviews were conducted from May 2017 to March 2018.

Staff education
The main focus of staff education was the implementation of strategies for increasing awareness and early recognition of sepsis, as well as the implementation of key elements of the updated Surviving Sepsis Campaign guidelines among all health care workers involved in care for patients with sepsis [21,22]. The study coordination bureau supported the local hospital quality improvement leaders by providing educative material (presentations, pocket cards, posters). Hospitals were also provided with a concept for a screening algorithm for the early detection of sepsis as well as recommendations for its implementation. Educational materials were provided for download via the website of the GQNS and concepts were presented at the annual meetings. The local quality improvment teams implemented education. In addition, ve web-based educational sessions were conducted between March 2017 and February 2018, recordings of these sessions were provided on the website of the GQNS. Due to overall low participation rates and technical problems reported by many participants, no further web based sessions were done. Further details on the interventions are provided in the study protocol (Supplemental Digital Content 1).

Evaluation of the effect of participating in the GQNS
The effect of participating in the GQNS was evaluated in a quasi-experimental difference-in-difference design [23]. A time-series analysis was conducted on the risk-adjusted mortality among patients with sepsis coded in participating hospitals. In this analysis the course and level of mortality during a retrospective baseline (January 2014 to March 2016) was compared to the course and level of mortalty during the intervention phase (April 2016 to June 2018). The comparison condition was the risk-adjusted mortality among all patients with sepsis in the national DRG-statistics during the same time periods. Changes in the evaluation concept as compared to the study protocol are presented in Supplemental Digital Content 3.

Outcome measures
The primary outcome was the monthly risk-adjusted hospital mortality per hospital of cases with primary or secondary hospital discharge ICD-10-GM code for sepsis with organ dysfunction including septic shock (R65.1, R57.2). Secondary outcomes were the risk-adjusted mortality among patients with septic shock (ICD-10-GM code R57.2) and among cases with sepsis and mechanical ventilation of more than 24 hours. Risk-adjusted mortality was calculated as risk-standardized mortality rate (RSMR, see Supplemental Digital Content 2 -De nition of variables, and Supplemental Digital Content 4calculation of risk-adjusted mortality). As the control condition, the German national DRG-statistics was used to calculate the monthly RSMR for all coded sepsis cases in Germany [24]. The difference between the monthly RSMR of each hospital in the GQNS and this national average was calculated and used as an outcome measure for the primary analysis. This controls for secular trends as well as seasonal variation. The development of the risk-adjustment model is described elsewhere [19].

Measures for intervention processes and implementation
To assess delity and extent of the local implementation of interventions in the participating hospitals, local quality improvement team leaders were surveyed at the end of the intervention period in autumn of 2018 with a standardized online questionnaire. The questionnaire contained items on the status of existing quality management structures, extent of usage of quality analysis and implementation of recommended interventions, as well as perceived barriers to change, and rating of the support provided in the GQNS.

Statistical analysis
Retrospective baseline (January 2014 -March 2016) and intervention phase (April 2016 -June 2018) were descriptively compared regarding patients' demographics, risk factors, the proportion of cases with mechanical ventilation >24 hours, hospital length-of-stay, and mortality. The monthly prevalence and RSMR were calculated and plotted to compare GQNS and the national DRG-statistics. To investigate the intervention effect on the monthly RSMR-difference between GQNS hospitals and the national DRGstatistics, interrupted time-series analyses by piecewise hierarchical models were calculated [25]. The start of the intervention was de ned individually for each participating hospital as the month of supply of the rst quality report. The intervention effect was tested by the signi cance of the change in the linear slope or change in level. Since small sample sizes of sepsis cases per month and hospital might cause bias by the unreliability of the RSMR estimate, the inverse of the noise-variance (see Supplemental Digital Content 4) of the RSMR were used as precision weights in a sensitivity analysis. Subgroup analyses were conducted among hospitals, which participated through the complete intervention phase, hospitals without complete participation, hospitals with ≤ 700 beds, and hospitals with > 700 beds. To analyze the processes of implementation of the intervention, descriptive statistics were calculated on the items of the survey of local quality improvement leaders.

Results
Forty-eight hospitals joined the GQNS before the distribution of the rst quality reports in April 2016 twenty-six additional hospitals joined during the intervention phase. Characteristics of participating hospitals are presented in Table 1. Characteristics of cases with sepsis are presented in Table 2. There were no relevant changes in demographics, comorbidities, or characteristics of the infection and sepsis. Hospital mortality was 43.5% during the retrospective baseline and 42.7% during the intervention period.

Tests of the effect of participation in the GQNS
The results of the interrupted time-series analysis on the difference between the RSMR of GQNS-hospitals and the RSMR from the national DRG statistics are presented in Table 3. There was no change in the trajectory of mortality for cases with sepsis across time before and after the intervention (percent change  Figure 1a presents the descriptive course of prevalence and RSMR for sepsis before and during the intervention period comparing participating hospitals in the GQNS and the national DRG-statistics; Figure 1b depicts the slopes and change in level calculated from the time-series analysis. Results of piecewise hierarchical models on the difference in the risk-standardized mortality rate (RSMR) between GQNS hospitals and the national German diagnosis-related-groups statistic. Slopes give the linear trajectory of RSMR-difference in % per month across time before and after start of the intervention, change in level gives the change at the time of the beginning of the intervention. Time of beginning of the intervention is de ned for each individual hospital as the time of supply of the rst quality report.
There were also no signi cant differences in slopes or changes in level in mortality among patients with septic shock or mortality among patients with sepsis and mechanical ventilation > 24 h, ( Table 3, Figure   1c to f). When precision weights were included to adjust for the unreliability of the RSMR estimated from small sample sizes, there were also no signi cant effects (data not shown).
When subgroups of hospitals were analysed, there was a signi cant change in slopes from baseline to intervention for hospitals, which participated through the whole intervention (p = 0.042, Table 4). While the difference to the national DRG-statistics showed a small increase during the baseline, there was no change across time observed anymore during the intervention, but there was also no decrease of mortality. No other subgroup showed any signi cant differences in slopes or level. Results of piecewise hierarchical models on the difference in the risk-standardized mortality rate (RSMR) in patients with sepsis between GQNS hospitals and the national German diagnosis-relatedgroups statistic considering different subgroups. Slopes give the linear trajectory of RSMR-difference in % per month across time before and after start of the intervention, change in level gives the change at the time of the beginning of the intervention. Time of beginning of the intervention is de ned for each individual hospital as the time of supply of the rst quality report. The overall rating of the support provided in the GQNS was good (median grade of 2 for work of the coordination bureau, as well as usefulness and usability of quality reports). implemented on ICU, normal wards, and emergency department 3 (6.1%) Descriptive statistics given as N (%) and median [1st quartile, 3rd quartile]. The survey was conducted among the local quality improvement leaders of participating hospitals in autumn of 2018 after the end of the intervention phase, one person per hospital was surveyed, since some local champions were responsible for more than one hospital, 69 participants were invited of which, 49 (71%) took part in the survey. a One participant did not provide information on this item. Rating of the support by the GQNS Grade for the work of the GQNS coordination bureau (1-6) 2 [1,2] Grade for usefulness of quality reports (1-6) 2 [1,2] Grade for usability of quality reports (1-6) 2 [2,2] Descriptive statistics given as N (%) and median [1st quartile, 3rd quartile]. The survey was conducted among the local quality improvement leaders of participating hospitals in autumn of 2018 after the end of the intervention phase, one person per hospital was surveyed, since some local champions were responsible for more than one hospital, 69 participants were invited of which, 49 (71%) took part in the survey.

Process evaluation
a One participant did not provide information on this item.

Discussion
The GQNS is a quality collaborative network using claims data and a complex risk adjustment to measure and improve the acute care quality for sepsis patients. Because of this pragmatic approach, 74 hospitals participated in the start-up period of the network. This evaluation study compared the development of risk-adjusted hospital mortality in cases with sepsis between the GQNS and the German national DRG-statistics in a quasi-experimental difference-in-difference design. It did not show an effect of participation in the GQNS.
The lack of success of the GQNS is not surprising since the majority of hospitals did not su ciently implement the recommended measures for quality improvement. Local quality improvement leaders reported high barriers to effective quality management -most importantly, lacking time and resources for quality improvement activities, as well as failure to generate hospital-wide improvement efforts due to general staff shortage and lacking involvement of all relevant departments and stakeholders. Similar reasons had been identi ed for the failure of the German cluster-randomized controlled MEDUSA trial, which comprised 40 hospitals and aimed to improve sepsis care by the establishment of change teams and prospective documentation and reporting of indicators of process and outcome quality, and staff education [10,11]. Likewise, the only published successful quality initiative on sepsis in Germany, by which an absolute reduction of mortality of 19% was achieved, received nancing and full support by the hospital's management board, which facilitated the hospital-wide role out of this program and the involvement of the crucial stakeholders [26].
The failure to replicate such successes in multicentre initiatives like the GQNS and MEDUSA indicate that quality improvement efforts on sepsis management in Germany should no longer be based solely on voluntary initiatives driven by intensive care physicians and nurses. Other high-income countriesnamely the UK, USA, and Australia -successfully implemented -speci c mandatory quality improvement indicators and tools on the national and regional levels during the last decade [27][28][29][30]. However, the stronger decline of sepsis-associated mortality in these countries compared to Germany [31][32][33][34] may depend not only on sepsis-speci c measures alone. Also more general tools for quality assurance and patient safety -such as rapid response systems, nation-wide education of health care workers in early warning scores for deteriorating patients, and the effective use of critical incidence reporting systems are mostly standard in these countries, but are poorly adopted in Germany [35][36][37][38][39]. German authorities and regulatory bodies in health care should follow these examples, become fully aware of the existing severe de cits in sepsis prevention and care, and take the necessary actions.
Besides the identi ed barriers to effective quality management, the failure to achieve substantial improvement might also be caused by aws in the approach taken by the GQNS. First, the GQNS only measured outcome quality in the form of risk-adjusted sepsis mortality, which alone does not give detailed insights into concrete possibly underlying care de ciencies [40]. Former successful quality initiatives on sepsis also used process quality indicators -primarily compliance to sepsis management bundles like timeliness of adequate antimicrobial therapy [9,26,28,41,42]. Additionally, benchmarking indicators of structural quality -like availability of in-house microbiological or standard operating procedures on antimicrobial treatment -could inform hospitals to implement concrete improvements.
Second, the GQNS relied on only using administrative claims data. This approach has high feasibility and low costs, but lacks the information necessary to de ne process quality indicators. Above that, several studies reported low validity for coding of sepsis, as well as risk factors for mortality -like comorbidities -in administrative data [43][44][45]. The low validity of the data might have impaired the usefulness of the quality reports to identify possible de ciencies of care and opportunities for improvement [12].
Automated surveillance may overcome these de cits to track sepsis rates and outcomes based on electronic health records [30], but can currently not be used among the majority of German hospitals due to the lacking implementation of electronic health records. Third, hospitals have repeatedly been shown to have major de ciencies in organizational and professional capacity to adequately learn and improve based on quality measurement [10,46,47]. Therefore, local quality improvement teams might bene t from education on change management, as well as concrete external guidance, support, and monitoring of the implementation progress as this is for example the case in the "SEPSIS KILLS" program of the Clinical Excellence Commissions of New South Wales in Australia [27].
The evaluation study of GQNS has several strengths. It was one of the few studies on sepsis-related quality initiatives using a controlled design [9,11,48,49]. In addition, a diverse sample representing the full spectrum of German acute care hospitals was included, which permits generalizing conclusions to the German health care system. The evaluation study also has limitations. It was based on claims data and might therefore be biased by changes in coding practices among participating hospitals across time.
The duration of the intervention phase of roughly two years might have been too short to result in observable changes [50]. We were only able to conduct six peer reviews during the intervention phase of the GQNS. The rst publication of the main quality indicators occurred in the summer of 2018, at the end of the intervention phase, and only by the 11 hospitals, which were obligated to do so since they had signed their contract for participation in 2015. It is unclear if a broader early implementation of these core elements of the intervention would have resulted in greater success.

Conclusions
The major reason for the failure of this voluntary initiative to improve sepsis outcomes in the majority of the hospitals was the lacking implementation of substiantial measures to improve early detection and adequate treatment of sepsis in the majority of hospitals. Major barriers to quality improvement were lacking time and ressources for quality improvement teams, general staff shortage, and a failure to involve all relevant stakeholders and departments in the quality improvement process. This indicates, that it was not possible to achieve adequate priority for sepsis among the pertinent stakeholders on the hospital board and department leadership level.
As long as the improvement of sepsis care does not receive adequate priority and resources in the German health care system, voluntary quality improvement initiatives are prone to failure. Sepsis needs to become a high priority for all health policymakers on the federal and national level. In particular, sepsis needs to be part of the list of mandated quality indicators for hospitals, and international standards for the early detection and the management of deteriorating patients need to be implemented to end preventable suffering from sepsis and reduce the burden for the German health care system.

GQNS
German Quality Network Sepsis RSMR risk standardized mortality rate Declarations Ethics approval and consent to participate The study was approved by the internal review board of the Jena University Hospital (IRB protocol 4536-08/15). Since a secondary analysis of claims data was done, the need for informed consent of patients was waved.

Consent for publication
Not applicable Availability of data and materials Only aggregated data per hospital were used for the evaluation of the study, these cannot be shared since contracts with the participating hospitals allow only the participating hospitals themselves to publish their respective quality indicators and prohibit that the study coordinators publish these data in any other form.
Competing interests DS and HR were funded in part by grants from the German Federal Ministry of Education and Research during and outside the submitted work and by annual fees paid by hospitals to participate in the GQNS.  Depiction of the effect of hospitals' participation in the GQNS. Panels a, c, and e present the descriptive changes in prevalence and risk-standardized mortality rate (RSMR) for patients with sepsis, septic shock, and sepsis with mechanical ventilation > 24 hours. Panels b, d, and f depict the slopes before and after the beginning of the intervention, as well as the change in level at the beginning of the intervention with