The CHANGE-3 study aimed at achieving a sustainable and largescale conversion of habits referring to the use of antibiotics for ARTI in ambulatory care. It was conducted as a two-armed cluster-RCT with a practice team intervention and a control group and a regional intervention which addressed the general public through a multi-media awareness campaign. A PE was conducted alongside the implementation program. In a complex intervention, a PE aims at understanding the functioning of the intervention by investigating the uptake of intervention components, mechanisms of impact and contextual factors [16]. The PE in CHANGE-3 focused on the effectiveness of the educational outreach visits plus a public campaign - as compared to public campaign only.
Study design
Outcomes of the program were assessed in a nested cluster-randomized trial, in which all practices and patients were exposed to the public campaign and practices in the intervention arm to educational interventions as well. Here we report on the process evaluation (PE) that was embedded in this study. The PE in CHANGE-3 was conducted as a prospective observational study. It used a mix of qualitative and quantitative methods in a convergent-parallel design.
In the cluster randomized trial, randomization and allocation of the 114 participating practices in the practice team study (BW n = 60, MV n = 54) to the two arms of the study was done by statisticians and concealed from all others in the project. Additional randomization of participants in the process evaluation was not applicable. Figure 1 illustrates the overall study design for CHANGE-3 with the highlighted process evaluation design.
Implementation program
The implementation program has been described elsewhere [17] and is briefly summarized here. General practices in the intervention group received an educational quality improvement program between September 2018 and March 2019 in order to strengthen health literacy competencies in GPs, MAs and patients. On single practice level, the intervention components addressed the healthcare provider team (GPs and MAs). A central component was an outreach visit by experienced OVs who presented general information on antibiotic prescribing and the CHANGE-3 project and discussed a feedback report on practice-specific antibiotic prescribing. Practice-individual feedback reports were generated from claims data provided by the statutory health insurance AOK and reflected prescribing from the fourth quarter in 2016. Additionally, all practices in the intervention group had access to an e-learning module focusing on strategies for provider-patient communication. During the outreach visit, all material referring to the regional intervention addressing the general public was to be introduced to the practice team.
The regional intervention consisted of a public awareness campaign addressing children, adolescents, parents, young and middle-aged adults as well as the older population with a multi-media approach using digital and analog information material (printed information material informing about antibiotics and the treatment of ARTI for patients, educational poster and flyer, coloring book, plush toy, comic for school-age children, educational magazine COLD, study-specific website and thematically focused e-learning delivered via a tablet application). Additionally, the feedback report detailing the practice individual antibiotic prescribing for ARTI and showing comparisons to other practice entities was to be supplied. To provide the opportunity for patients to view multi-media content of the regional awareness intervention in the GP practice, all participating practices in the intervention group received a tablet computer to be placed in their waiting area. Tablet-based apps have been used in diverse contexts to provide educational information [18–20]. GP practices in the control group had access to all analog material used in the awareness campaign and could order supplies of those through the study-specific educational website. The regional awareness intervention ran between September 2018 and January 2020. Table 1 lists all components offered to participants in the CHANGE-3 study. Figure 2 displays the chronology of study components and data collection periods of the process evaluation.
Table 1
Intervention components offered in the CHANGE 3 study
Category | Intervention group | Control group |
Analog components | | |
for GPs, MAs and patients | Flyer* (German and other languages) | Flyer* (German and other languages) |
| Poster | Poster |
| COLD magazine** | COLD magazine** |
| Plush toy** | Plush toy** |
| Comic** | Comic** |
| Coloring book | Coloring book |
for healthcare provider teams | Outreach visit*** | |
| Feedback report (two times) | |
| Printed thematic background information via postal mail | Printed thematic background information via postal mail |
Digital components | | |
for GPs, MAs, patients and general public | Informative website | Informative website |
for patients | Tablet devices providing educational app | |
for healthcare provider teams | e-learning platform providing communication training | |
*Available in German, English, French, Turkish, Vietnamese, Russian, Arab |
**Component only offered in second half of the intervention period |
*** Component only offered in first half of the intervention period (one time) |
Study population
Based on a two-wave survey for GPs and MAs and qualitative interview research with GPs, MAs, OVs and patients, the process evaluation focused on physicians and MAs allocated to the intervention group of the study, their outreach visitors (OV) and patients, as well as GPs, MAs and patients in the control group. The intervention in the CHANGE-3 study (educational practice team study) was intended for 114 participating general care practices in Baden-Wurttemberg (n = 60) and Mecklenburg-Vorpommern (n = 54) [17].
To be eligible for participation in the PE, ambulatory practices needed to be located in Mecklenburg-Vorpommern or Baden-Wuerttemberg, allocated to the intervention group or the control group of the practice team study of CHANGE-3, and belong to the medical specialist group of General Practitioner. MAs eligible to participate in the process evaluation were employees of participating practices. Patients eligible to participate in the PE were patients of the participating general practices who had been treated in said practices for respiratory tract infection during the intervention period. Further inclusion criteria were written and spoken German language skills and > 18 years old. Patients living in care facilities, suffering from dementia, incapacitated mentally or contacting a practice during regular visitation hours in locum care were excluded. Patients meeting the specified inclusion criteria were approached in the GP practice by the local healthcare providers. OVs eligible to participate in the PE had carried out at least five outreach visits to participating practices. All participants in the process evaluation had to be aged 18 years and above with full command of written and spoken German. Written informed consent was considered a pre-requisite for participation and were obtained separately for survey and interview participation during the recruitment process.
Due to the drop-out of 5 practices, data could be collected in 109 participating practices. All physicians (n = 132) and a proportion of MAs in the intervention group and in the control group contacted for general study participation received information about the PE and were invited to participate in it. MA participation was limited to two per practice with the intention to limit imbalance in the sample and to stay within the available reimbursement budget.
The PE followed a purposive sampling strategy with regard to region and sex to recruit telephone interview participants. Recruitment was supported and initiated by the aQua Institut, Goettingen, and carried out by the study team at the Department of General Practice and Health Services Research, University Hospital Heidelberg. Using an opt-in approach, participating practices in the intervention and in the control group were asked to support patient recruitment along a structured process. The practice team handed out written information about the study and the PE, as well as a contact form to be sent to the Department of General Practice and Health Services Research, University Hospital Heidelberg via fax by the respective practice team. OVs who had conducted a minimum of five outreach visits each were contacted through the study team supported by the aQua Institut, Goettingen.
All potential interview participants received written information about the PE, as well as a contact form to be sent by fax or e-mail to the study team. After interest in participation in the PE was stated via the initial contact form, the informed consent form was mailed out. Members of the study team contacted interested GPs, MAs, patients and outreach visitors by phone to provide detailed information verbally. Upon receiving the informed consent form signed by the recruited participant, interview dates were scheduled.
Measures
The survey and the interview guides were based on the Theoretical Domains Framework (TDF) [21, 22] and focused on aspects regarding the regional intervention with multi-media content and e-learning, the practice team intervention with outreach visit and feedback report and on personal perceptions regarding the provision of health services to patients with ARTI. Survey and interview study included questions referring to socio-demographic aspects like age, gender, years of working experience and characteristics of the working environment.
Tailored questions for the interviews and survey questionnaires covered (a) the uptake and perceived impact of intervention components by participants with a focus on care for patients with ARTI, (b) intervention effects on daily practice as well as on decision support, (c) the perceived impact of diverse context factors on health services for patients with acute respiratory tract infections, and additionally, the written questionnaires contained items about (d) perceptions of patients’ expectations regarding antibiotics prescribing.
Survey study
Data was generated via a two-time survey (GPs, MAs) across the intervention and control group. Survey periods were April to July 2019 (T1) and March to April 2020 (T2). The T1 questionnaire had a total of 5 sections. Section A focused on aspects regarding the regional intervention with multimedia content (web-based public campaign). Section B related to the practice team intervention with outreach visit, e-learning and feedback report (intervention group only). Section C comprised items referring to contextual factors (structural practice factors; motivational factors for prescribing decisions). Section D focused on personal attitudes and perceptions regarding the provision of health services to ARTI patients. In Section E, socio-demographic and practice characteristics were asked for (gender, age (year of birth), professional experience, size and location of practice (number of patients per quarter; urban, rural), participation in continued training, type of practice (single or group-practice, health centre, other). The T2 questionnaire comprised 3 sections. All items in section A referred to the intervention components. Section B focused on attitudes and perspectives on patient care and section C asked for a subset of the socio-demographic and practice characteristics (year of birth, gender, professional experience and number of patients seen per quarter). The questionnaires for MAs were adapted in section D and asked about patient expectations. T1 and T2 questionnaires for the control group followed the same structure, but contained no items referring to the practice team intervention.
Written, paper-based questionnaires were mailed to participating GPs and MAs across study groups at two separate points in time (T1, T2). To increase the response rate, a reminder was sent out for T1 three weeks after the mailing date. Survey items were scored on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Questionnaires were returned in an enclosed postage-paid envelope directly to a study nurse at the Department of General Practice and Health Services Research, University Hospital Heidelberg who registered and pseudonymized them. Data generated via the paper-based questionnaires were transformed into electronic data sets.
Interview study
In the intervention group, semi-structured guide-based telephone interviews were conducted with GPs, MAs, patients and OVs. In the control group, interviews were exclusively applied to patients. In depth interviews were executed in two phases. In the first phase, GPs, MAs and patients were interviewed regarding the overall objectives of the PE. In the second phase, OVs, GPs and one expert in the field were interviewed regarding their perceptions about outreach visits.
All interview guides were developed by the interprofessional team of researchers (Health Services Research, Public Health, General Practice) (see Additional files 1–4 for translated versions) and were based on constructs of the TDF, a literature review and pre-defined research questions. All interviews were audio-recorded, pseudonymized and transcribed verbatim.
Data analysis
All data collected from the survey, the interview study, and socio-demographic questionnaires were pseudonymized prior to analysis and electronically saved and stored on secure servers at the Department of General Practice and Health Services Research, University Hospital, Heidelberg.
In the survey study, descriptive statistics were used to (a) characterize the study sample, (b) assess fidelity and perceived impacts on daily practice of healthcare delivery, and (c) impact of context factors on health services for patients by tabulating measures of the empirical distribution. According to the level of variables, means, standard deviations (SDs), and absolute or relative frequencies are reported. The general uptake of devices is reported by transforming ordinal five-point Likert scale levels into binary variables. The statistic software IBM SPSS 25 (IBM, Armonk, NY, USA). was used to conduct quantitative analysis.
In the interview study, all interviews were audio-recorded and transcribed verbatim. Thematic framework analysis was used to classify and organize data deductively according to a priori defined categories derived from the TDF [21] to identify determinants of practice which influence health care delivery to ARTI patients in general practices, with regard to the educational components of the intervention and with regard to converting habits of antibiotic use. Categories were also defined inductively de novo according to concepts and key themes that emerged from the data. Data were managed, analyzed and coded by junior and senior researchers of the study team (with backgrounds in Public Health and Health Services Research) using qualitative data analysis software (MAXQDA, 2018.2). All researchers involved had prior experiences with qualitative methods. The pre-defined categories of the TDF were used to identify determinants of practice which influence healthcare delivery to patients with acute respiratory infections in general practices with regards to the educational components of the intervention and with regards to converting habits of antibiotic use.