Fidelity as a term describes the level to which an intervention was delivered as intended . In this respect, the most commonly practiced framework  distinguishes between adherence and moderator domains to provide a cause-effect-principle explaining fidelity measures. Following this comprehension, adherence is defined as a bottom-line measurement describing the dose and content of an implementation program. If an intervention completely adheres to a study protocol, fidelity can be rated high. To understand mechanisms affecting adherence scales, factors that affect the level of fidelity need to be identified. In this study, these factors originated from five self-reported domains describing participants’ perceived views on implementation. Figure 1 comprises the elements of adherence and considered domains. Since the framework on implementation fidelity has continuously been extended, this analysis included the additional domain of ‘context’ introduced by Hasson . The characterization of dose in quality improvement measurements provided by McHugh et al.  was included into the theoretical model for this present study.
Study design of the ARena trial
The ARena implementation program was designed as a three-armed, non-blinded cluster randomized trial with an added cohort reflecting standard care. Randomization was performed by the Institute of Medical Biometrics and Informatics at the University Hospital Heidelberg. The implementation program was organized by the aQua Institut, Goettingen, and embedded into 14 primary care networks (PCN) in two federal states (Bavaria and North Rhine-Westphalia) in Germany. PCNs are regional associations of primary care practices aiming at facilitating quality improvement initiatives, representing interests at health insurance companies as well as reimbursing additional activities for member practices . In order to understand the role of PCNs in the dissemination of the implementation program, this level of randomization has been chosen for primary outcome analysis. The implementation program consisted of different components applied to each of the three study arms. Arm I received a standard set comprising a public information campaign, patient information material, e-learning addressing physician-patient communication, thematically relevant QCs (common respiratory tract infections (CRTI), urinary tract infections (UTI), community acquired pneumonia (CAP), multi-resistant pathogens (MRP)) containing data-based feedback for physicians, and the performance-based bonus. Arm II received the standard set plus e-learning modules addressing patient communication and QCs targeting non-physician health professionals as well as patient information material provided via tablet devices. Arm III received the standard set, a computerized decision support system integrated in existing practice management software and multidisciplinary QCs in local groups. All participating practices could receive reimbursement for project-related expenditure. A detailed display of the study design is provided by the study protocol .
The intervention period encompassed 21 months. In total, 196 practices with 312 physicians and 99 medical assistants (MA) participated. The statutory health insurer AOK provided routinely collected claims data referring to consultations for non-complicated infections in the intervention arms and the added cohort reflecting standard care in Bavaria and North Rhine-Westphalia. The study design and detailed sample size descriptions of the ARena trial are illustrated in Additional File 1, Supplementary Fig. 1.
Study design of the process evaluation
The ARena trial was accompanied by a process evaluation (PE) which intended to understand working mechanisms affecting primary and secondary outcomes as well as determining the level of fidelity to the program . The PE was designed as a prospective observational study and conducted with a mixed methods approach containing a longitudinal survey study and an interview study. The survey study consisted of written questionnaires targeting participating physicians of study arms I, II and III and participating MAs of study arm II. For each intervention arm, a tailored questionnaire was developed. Data collection took place at three different points in time (T0-T2). The interview study targeted participating physicians, MAs and stakeholder representatives of PCN managements, health insurance providers, the association of statutory health insurance physicians and self-help organisations. Additionally, implementers documented overall participation over the course of the study, utilization of e-learning and CDSS, attendance to QCs, and reimbursement for project- and patient-related expenditures. This present study was based on survey data collected during the PE and the additional documentation (attendance data). Findings of qualitative analyses have been reported elsewhere [37–39]. Figure 2 summarizes the study design and sample size of the PE.
An extensive description of the study population of the ARena trial is provided in the protocol . To be eligible for participation in the PE, practices needed to be enrolled in one of the 14 participating PCNs and had to be allocated to one of the three intervention arms. Physicians had to represent one of the medical specialist groups of general practitioners, internists, gynecologists, ear-nose-throat specialists, urologists, pulmonary specialists or pediatricians. MAs eligible for participation in the PE were employees of participating practices. Across participant groups, further inclusion criteria were written and spoken German language skills, 18 years of age or older and a written declaration of consent to participate in the study. No additional exclusion criteria were assigned.
Recruitment and sampling for the survey study
The PE followed a voluntary response sampling strategy. By signing the consent form of the ARena trial, participants also consented into participating in the PE. The Department of General Practice and Health Services Research at the University Hospital Heidelberg compiled a cover letter and written information material detailing the procedures and aim of the PE. The Department’s ARena study team of researchers developed the study-specific survey questionnaires based on the Theory of Planned Behaviour . The questionnaire items were used to gain insights into the impact of the intervention components and contextual factors . The aQua Institut (Goettingen) led the project and thus contacted enrolled practices and sent the survey questionnaires by mail. After four weeks, e-mail reminders were sent to increase response rates.
Data collection and measures
Participants’ views on the implementation and the engagement in key components of ARena (QCs and additional bonus compensation) were explored in a self-reported questionnaire. Questionnaires were dispatched to participants in January 2018 (T0), October 2018 (T1) and July 2019 (T2). All questionnaires focused on adherence to intervention components and views on the implementation. T1 and T2 questionnaires additionally asked for intermediate and final conclusion regarding the assessment of intervention components. Completed questionnaires were returned and registered by the ARena study team at the Department of General Practice and Health Services Research, University Hospital Heidelberg, between February and April 2018 (T0), November 2018 to January 2019 (T1) and July to September 2019 (T2). Received questionnaires were digitalized and transferred into IBM SPSS Statistics 24. Survey items included for each domain of participant views are listed in Additional File 2, Supplementary Table 1.
The participants’ views on the implementation were measured in five domains: 1) ‘Participant responsiveness’ contains items regarding the respondents’ perceptions about the usefulness of components and their potential to facilitate new impulses in the context of rational antibiotics use. 2) ‘Quality of delivery improvement’ refers to reflections about the extent to which the ARena participation supported guideline-oriented prescribing patterns and fostered security in therapeutic decisions. 3) ‘Contextual facilitators’ refer to the role of PCNs in optimizing patient care as they were seen as a major design element of the ARena study. 4) ‘Positive antibiotic attributions’ consider physicians’ perceptions about positive ancillary effects of antibiotic use such as reduced consultation time. 5) ‘The culture of shared decision-making’ (SDM) score reflects the respondents’ integration of patient- and peer-views into therapeutic decisions.
Attendance and use of financial bonus
Adherence to QCs, e-learning, CDSS, and basic expenditure reimbursements were identified by documented attendance data. Triggered additional bonus payment was identified from the claims data. Overall, data of 196 practices were collected by the aQua Institut over the intervention period of 21 months between October 2017 and June 2019. Variables regarding participant attendance of QC meetings are reported on practice level and were collected in the respective events. Attendance data was documented in Microsoft Excel 2019 and subsequently transferred to IBM SPSS Statistics 26. Variables providing information about additional bonus compensation were collected using the claims data aggregated on practice level.
Adherence was subcategorized in the domains of content and dose. Indicators representing content were exclusively collected for the additional bonus compensation component. Indicators representing dose which was further split into domains of exposure and engagement were collected for both, QCs and additional compensation components.
Based on the survey and attendance data, the intervention fidelity was explored. Indicators were developed to map the participants’ engagement in five intervention components. The descriptive analysis explored absolute and relative frequencies on physician and practice level in the intervention arms. Sociodemographic factors, adherence data and participants’ views on implementation were analyzed descriptively. For continuous variables, means, medians, min/max and standard deviations are provided, for categorial and ordinal variables absolute and relative frequencies are reported. Survey items were based on a 5-point-Likert scale ranging from “Strongly Disagree” to “Strongly Agree”. Items representing one domain of participants’ views on implementation were scored using mean value calculations and tested on internal consistency using Cronbach’s Alpha procedures. Binary correlations between dependent and independent variables were determined by calculating Pearson and Spearman correlation coefficients.
A binary logistic regression model was used to identify directional coherence between the engagement in additional bonus compensation representing the outcome variable and the five domains of participant views on implementation representing predictor variables. A multiple linear regression model was computed regarding association between engagement in the four QC themes reflecting the outcome variable and the five domains of participant views reflecting predictor variables. Predictor variables were considered on a metric scale level and tested on multicollinearity with a set threshold of r ≥ 0.7. Missing values were marked accordingly and excluded from analyses. Effect sizes were reported by Odds Ratios (OR) and Beta coefficients including 95% confidence intervals. To provide information about data accuracy, confidence intervals, standard errors and the coefficients of determination R2 are listed. All models have been adjusted by age, sex and intervention arm affiliation. Additional multilevel analyses were conducted considering a hierarchical data structure of practices representing the random effect (MIXED and GENLIN estimations). Due to a high loss of cases within data linkage efforts between attendance and survey data, outcome variables of estimation models based on self-reports of the T2 survey only. The level of significance was set at p ≤ 0.05. Since this is an explorative analysis all p-values are of descriptive nature.