Overall Study Design and Setting
The risk prediction model of interest is ACCEPT (ACute COPD Exacerbation Prevention Tool), a clinical prediction model for 12-month risk of COPD exacerbations. While there are several clinical prediction models for COPD exacerbations, a systematic review preceding the publication of ACCEPT concluded that “none of the existing models fulfilled the requirements for risk-stratified treatment to personalize COPD care”, due to the inability of these models to make truly individual-specific predictions, and the absence of rigorous validation (14). ACCEPT was designed specifically to address the shortcoming of previous studies, was extensively validated, and is considered the first COPD exacerbation clinical prediction model to be ready for clinical practice implementation (15).
The IMPACT study will be implemented in two subsequent phases at two University of British Columbia (UBC) teaching respiratory clinics in Vancouver, BC, Canada: St. Paul’s Hospital and Vancouver General Hospital. During phase one, we will conduct research with users to design an intervention that consists of a pulmonologist decision support tool that integrates the ACCEPT clinical prediction model into the EHR platform, and an accompanying patient informational handout. When phase one is completed, we will conduct a stepped wedge randomized controlled trial (RCT) to evaluate the impact of the intervention on the process of care, and on both patient-reported and clinical outcomes. The specific study design, population, and procedures for each of the two study phases are described separately in the following sections.
Phase One: Development of Decision Support Tools for Pulmonologist and Patients
Study Overview
Based on initial conversations with treating pulmonologists and patient partners, two tools will be developed to aid in the implementation of ACCEPT at point of care: 1) a decision support tool for pulmonologists, called the ACCEPT Decision Intervention (ADI); and 2) a printed informational handout for COPD patients (‘patient tool’). Development of the ADI and the patient tool will follow the methods put forward by the International Patient Decision Aids Standards collaboration for systematic and user-centered development of decision aids (16, 17). The development process is iterative, involves consultation with stakeholders throughout, and will be undertaken with integrated knowledge translation specialists in alignment with both the Knowledge-to-Action process model (17, 18), and the theory and frameworks comprised within the Theory and Techniques Tool (19, 20). Tool development will be divided into four related but distinct steps: 1) information gathering; 2) prototype building; 3) qualitative evaluation; and 4) pilot testing (Fig. 1).
Study Population
Phase one study participants will fall into one of three groups: 1) pulmonologists, 2) individuals with COPD, and 3) clinic staff. Individuals with COPD will be eligible to participate if they have a diagnosis of COPD, are at least 18 years of age, and are able to speak and provide consent in English.
Tool Development Process
To ensure that the tools are relevant to the intended users, initial work will identify the decisional needs and preferences of pulmonologists, and people with COPD, via semi-structured interviews. These decisional needs and preferences will inform the content and format of both tools. The interview guides will be developed collaboratively with subject area experts (e.g., decision scientists, specialists in smoking cessation) and intended tool users (e.g., pulmonologists, people with COPD). Interviews will be audio-recorded and interview notes will be analysed to identify preferred structure and content of the two tools. Additional interviews will be conducted with clinic staff to assess the clinic workflow and assist in tool implementation. Following the interviews, the initial prototype tools will be developed using a collaborative process to draw on the unique skillsets and perspectives of different team members. At the center of this process will be decision scientists, integrated knowledge translation specialists, and COPD patient partners. The content and format of the tools will be derived from the semi-structured interviews, and informed by the Theory and Techniques Tool (19, 20). The Theory and Techniques Tool is an interactive resource that links specific behaviour change techniques with mechanisms of action. It will be used to ensure that content is delivered in an evidence-based manner. Specific content will be developed with subject area experts on the study team (e.g., pulmonologists) to ensure accuracy and alignment with best guidelines and practice principles. Patient partners and pulmonologists will provide input on the content areas and formatting before prototypes are developed, as well as provide feedback on prototypes to ensure the tools meet user needs and are easy to use and understand.
The prototype tools will be revised and refined through cognitive interviews (21). Participants will be the intended tool users (i.e., pulmonologists will be interviewed for the ADI; and people with COPD will be interviewed for the patient tool). During the interviews, the relevant prototype tool will be presented to participants who will be instructed to navigate through the tool while stating their thoughts aloud. As needed during the interview, the interviewer will prompt the participant on specific aspects of the tool identified as important by the research team, or previous participants. Interviews will be designed to elicit feedback on the layout, understandability of the language used, appropriateness of the content, and ease of use. When applicable, participants will be presented with previous versions of the relevant tools to assess the acceptability of potential changes based on previous participants’ suggestions. Cognitive interviews will continue until saturation is reached. For this work, saturation will be defined as no new significant concerns arising during cognitive interviews.
Following cognitive interviews, a preliminary evaluation of the patient tool will be conducted. The metrics with which the tool will be evaluated will be determined in consultation with patient partners. Potential metrics include: knowledge, system usability (22), acceptability (23); and decisional conflict (24). Quantitative evaluation of the ADI may be conducted after consultation with the study team. This evaluation would focus on usability and acceptability of the ADI.
Phase Two: A Stepped-Wedge Cluster Randomized Control Trial
Study Design and Setting
To test the effect of the intervention package (ADI and patient tool), we will conduct a prospective stepped-wedge cluster RCT. The unit of randomization will be staff pulmonologists at the two study sites. A stepped-wedge trial design was chosen for multiple reasons. First, clinical prediction model implementation studies should not randomize patients because of potential learning effects (changes in physician behavior after exposure to prediction model, even towards patients in the other group) (25). Second, unlike a conventional cluster RCT, the staggered treatment assignment in this design will provide opportunities to control for temporal trends. Finally, the stepped wedge RCT is an implementation-oriented study design, since by the end of the trial all clusters will be assigned to the intervention arm.
In this trial, pulmonologists (clusters) will be assigned to treatment-as-usual (comparison arm) at the beginning of the study, with successive assignment to the intervention arm completed in a one-directional, staggered format. The one-way crossover from comparison to intervention will be allocated using a computerized random number generator and completed by the project manager during the first month of the study (phase-in period). Further, due to the nature of the intervention, masking is not possible, thus no methods to conceal the allocation sequence nor blinding will be applied at the patient or cluster level. Key study design features are shown in Fig. 2.
Study Population
Since the intervention will be embedded into the EHR as a standard clinical decision support system and will be part of routine care, all staff pulmonologists at the two study sites (n = 28) will be automatically part of the study. However, conservatively anticipating data from the practice of four physicians will not be used for this study for various reasons, such as an extended leave, we aim to use data from 24 pulmonologists, who will comprise the study clusters. Individual patients who are referred to one of the participating pulmonologists at a study site with a physician-assigned diagnosis of COPD will be invited to participate in the clinical trial. For inclusion, individuals must be: 1) a legal Canadian resident; 2) aged 18 years and older; and 3) able to understand English.
Intervention
The study intervention has two related components that will be developed during phase one: 1) the ADI, which is a software decision tool integrated within the EHR platform; and 2) a patient tool, which is a tailored educational handout for COPD patients (Fig. 3). The ADI combines risk prediction from ACCEPT with the CTS definition of low- versus high-risk (for exacerbations). Features of the ACCEPT clinical prediction model, including details of the development and validation studies, are reported elsewhere (26). In summary, ACCEPT uses routinely collected patient characteristics to predict the rate and severity of exacerbations in patients with COPD during the next 12 months. Predictors include sex, age, weight, height, smoking status, symptom burden, lung function, history of exacerbations, cardiovascular risk (represented by the use of statins), and current COPD therapies. Several of these predictor values will be prepopulated from the electronic patient chart, but pulmonologists will be given the option to override these values. To be aligned with the CTS guidelines, the frequent exacerbator group will be defined as those with predicted moderate exacerbation rate ≥ 2 per year or predicted severe exacerbation rate ≥ 1 per year.
The second component (the patient tool), will be a short, individualized patient information pamphlet. The information included in the pamphlet will be determined during phase one and will be tailored to each patient’s recommended treatment based on their ACCEPT score, their clinical needs, and their lung function.
Study Outcomes
This is a complex trial and we hypothesize that the overall benefits of intervention will be multifaceted, including changes in the rate of under- and overtreatment, adherence to treatments, reduction in exacerbation rates, and narrowing of the known sex- and gender gaps in COPD care (27, 28). Table 1 describes the study’s primary endpoint and secondary outcomes, and how each of these measures will help us to evaluate our hypotheses. We chose ‘prescription appropriateness’ as the primary endpoint because it is the process of care outcome that is most directly affected by the intervention, and it has been frequently used in trials on computerized decision support tools (29). Prescription appropriateness will be assessed by determining the concordance between the pulmonologist- and CTS-based treatment recommendations, determined through an independent patient interview and chart review by a research coordinator after each clinic visit.
Table 1
Description of study outcomes and corresponding hypotheses evaluated by the clinical trial.
Outcome label
|
Outcome
|
Outcome Measure
|
Relevant hypothesis
|
Rationale for selection
|
Primary outcome
(Outcome 1)
|
Prescription appropriateness
|
The clinician prescription will be considered concordant if it is the same as the prescription based on the ACCEPT recommendation (or if ACCEPT suggests more than one eligible prescription, the clinician prescription is one of them), otherwise it will be considered discordant.
|
Reduction in under- and overtreatment
|
Reclassifying individuals from high- to low-risk will avoid expensive therapies in patients who will not benefit from them, while reclassification from low- to high-risk will result in more intense therapies that will reduce exacerbation risk.
|
Outcome 2
|
Medication adherence
|
Medication adherence will be assessed in two ways: a) using the Medication Possession Ratio, and b) self-reported medication adherence. (a) Medication Possession Ratio is defined as the ratio of the total days' supply dispensed to the total days' supply prescribed during the study period. (b) Self-reported adherence will be assessed using the COPD-specific Beliefs about Medicines Questionnaire (BMQ)(30). The COPD BMQ is a 15-item scale with 2 subscales assessing patients’ perceptions of their need for medicine and concerns regarding medicine.
|
Higher adherence of patients to treatment recommendations
|
We hypothesize that accurate risk communication will improve the adherence of patients to the recommended treatment (29,31).
|
Outcome 3
|
Rate of moderate/severe exacerbations
|
Moderate exacerbations: any outpatient physician visit for COPD followed by filling prescriptions for antibiotic or oral corticosteroids.
Severe exacerbations: a hospital admission with the main discharge code of COPD.
|
Change in rate of moderate / severe exacerbations
|
The departure from guideline-base care is a mixture of over- and under-treatment, and the proposed intervention is likely to reduce such departure from guidelines. Correcting under-treatment will reduce exacerbation rate, while correcting overtreatment will not materially change the rate, resulting in a net reducing effect. In addition, improving adherence will further reduce exacerbations.
|
Outcome 5
|
Impact of COPD on health status
|
Measured by the COPD Assessment Test (CAT)(32). The CAT includes 8 questions scored from 0 to 5, with higher scores indicating more severe impact of COPD.
|
Improved health status
|
Patient-reported outcomes are likely to be affected by a quality-improvement intervention that involves addressing low health literacy and inaccurate medication beliefs (33).
|
Outcome 6
|
Quality of life
|
Euro Quality of Life – 5 Dimensions (EQ5D). The EQ5D is a standardized questionnaires that assess 5 dimensions of health status, each at 3 distinct levels (34). These data will be used to track changes in patient-report quality of life over the duration of the study.
|
Improved quality of life
|
Patient-reported outcomes are likely to be affected by a quality-improvement intervention that involves addressing low health literacy and inaccurate medication beliefs (33).
|
Outcome 7
|
Smoking cessation
|
(a) Motivation will be assessed by asking patients to classify themselves into one of the following 3 stages: a) intending to quit smoking; b) intending to quit within 6 months but not within 1 month; and c) not intending to quit within 6 months.
(b) Nicotine dependence will be assessed using the Fagerström Test (35)—a questionnaire made up of 7 questions, the first scored as 0 and 1 (yes/no), and the remaining 6 multiple choice questions scored from 0 to 3. The total test score ranges from 0 to 10, with higher scores indicating more intense physical dependence on nicotine.
|
Improved lifestyle behaviors
|
While smoking cessation is recommended for all current smokers with COPD, very few are offered such an intervention. Studies have suggested that even a brief intervention can be effective on reducing exacerbation rates (36).
|
ACCEPT, Acute COPD Exacerbation Prevention Tool; BMQ, Beliefs about Medicines Questionnaire; CAT, COPD Assessment Test; COPD, chronic obstructive pulmonary disease; EQ5D, EuroQoL 5-Dimension. |
Secondary clinical outcomes include medication adherence (Outcome 2) and rate of moderate or severe exacerbations (Outcome 3). These clinical outcomes will be assessed by linking patient data to BC’s administrative health databases using a unique Personal Health Number. These databases comprehensively capture all healthcare encounters of all legal residents in BC at an individual level, independent of payer (37). Medication adherence will be assessed using the Medication Possession Ratio. For Outcome 3, we will apply a validated definition of moderate and severe exacerbations; moderate exacerbations will be defined as any outpatient physician visit for COPD followed by filling prescriptions for antibiotic or oral corticosteroids, and severe exacerbations will be defined as any hospital admission with the main discharge code of COPD (38).
For patient-reported outcomes, we will assess self-reported medication adherence, quality of life, and smoking cessation using follow up questionnaire data. Self-reported medication adherence will be assessed using a specific version of Beliefs about Medicines Questionnaire (BMQ -COPD), which has been shown to have excellent construct validity and to be a strong predictor of adherence behavior (39). For determining the impact of the intervention on the patients’ health status and quality of life, we will administer two questionnaires: 1) the COPD Assessment Test (CAT), and 2) the EuroQoL 5-dimension (EQ5D). The CAT assesses the impact of COPD symptoms on a person's daily activities (32), and the EQ5D is a validated tool used to measure a patient’s overall quality of life (34). Smoking cessation will be assessed by determining patients’ barriers to cessation through 1) motivation, and 2) physical nicotine dependence.
For the sex and gender effects, we will address two a priori hypotheses. First, we hypothesize that gender-related effects are, over and beyond biological sex, independent predictors of exacerbation risk and their incorporation into risk stratification can improve predictive accuracy. Second, the extent of gaps in COPD care is known to be different between men and women (28), and we hypothesize that incorporating the decision tools into clinical care will change such sex/gender gaps. Our study includes novel data collection on gender identity, institutionalised gender, including socioeconomic status that will be assessed during the initial study visit (Supplementary Material – Section 1).
Study Procedures
Recruitment, Enrollment, and Consent
At the beginning of the study, the project manager will review key information from the study protocol with participating pulmonologists. Pulmonologists will also be emailed the latest CTS guidelines (6), and be provided one-on-one refresher training on the guidelines, if requested. Participation in these orientation interactions, however, will not be mandatory and lack of participation by a pulmonologist will not exclude them from the study.
For patient recruitment, clinical research coordinators at each site will review the clinic schedule and related patient charts every morning to identify potentially eligible COPD patients under the care of participating pulmonologists. At the end of each appointment, the site coordinators will approach eligible patients to invite them to participate in the study and obtain informed consent. If the pulmonologist is assigned to the intervention arm, a coordinator will set up software triggers for all eligible patients to deliver the intervention. A trigger will launch when the pulmonologist opens a patient’s EHR chart during the patient encounter and the ADI will be pre-populated with predictor values. If a predictor value is not retrieved from a patient’s EHR, the empty fields will be highlighted, and will be fillable through dictation or by typing directly into the ADI computer interface. At the end of the encounter, the pulmonologist will print the customized patient tool that will be distributed during the post-visit interview with the research coordinator.
In line with the principle of pragmatism, patients can consent to participate in the cross-sectional component alone (which is sufficient for the assessment of the primary endpoint), or to both the cross-sectional and longitudinal components of the study for assessment of the primary endpoint and secondary outcomes (Fig. 4). The cross-sectional component consists of a post-visit interview with the patient and research coordinator during which the coordinators will record 1) the ACCEPT score and corresponding recommended treatment based on the CTS guidelines; and 2) the pulmonologist prescribed treatment. These data will be used to determine which treatment the patient should have received (prescription appropriateness).
If the patient consents to participate in the longitudinal component, the initial visit will include both an assessment of the recommended and prescribed treatment, and administration of questionnaires to collect clinical and patient-reported data to assess the secondary outcomes (Fig. 4). Patients in the longitudinal component will be followed up at three- and six-months post initial visit via telephone to administer the relevant questionnaires. If a patient sees an enrolled pulmonologist multiple times throughout the 24-month study period, the first visit with the pulmonologist will be used to mark the start of the follow up period for longitudinal data collection, unless the pulmonologist was reassigned to the intervention arm between visits. In this case, if the patient has consented to the longitudinal component, the follow up period will be reset during the first visit the pulmonologist is in the intervention arm, and the patient will be followed for another six months after the initial intervention arm visit.
Data Collection
The total duration of the trial is 30 months. Initially, all participating pulmonologists will be in the comparison arm. After six months, two randomly selected pulmonologists will be reassigned to the intervention arm every month. There will be a one-month phase in period for all pulmonologists at the start of the trial, with patient recruitment and data collection starting in month two. In month 18, the last two pulmonologists will be assigned to the intervention arm and patient recruitment will continue until month 24. Follow-up data will be collected until month 30 to ensure six months of follow-up data for all patients. This allocation scheme was devised to strike a balance between statistical efficiency and the logistical demand of switching pulmonologists from the comparison arm to the intervention arm.
Data Monitoring and Management
Before trial initiation, we will establish a Data Safety and Monitoring Committee (DSMC) to ensure that the rights and overall well-being of participants are safeguarded, and attend to any immediate safety matters. Members of the DSMC will be selected by the study investigators but will act fully independently from the study, its investigators, and the funder. The DSMC will consist of individuals with methodological and clinical expertise.
Throughout the trial, study personnel will be available during to troubleshoot and address questions about ADI use and triggers. Biweekly reports will be generated to track the frequency of ADI triggering, ADI completion, and proportion of recommendations overridden. This will allow us to determine the fidelity of ADI usage. If use of the ADI is lower than expected for a pulmonologist, we will approach them to determine potential issues.
All research data will be entered daily by the clinical research coordinators at each study site. We will use an electronic case report form for direct data entry into REDCap®. This minimizes data entry error and improve data quality through the immediate application of data validation rules and boundary checking. Each record will be assigned a unique subject ID that will not include any personal or identifying information. A master list linking identifiable participants to research data will be kept encrypted and password-protected under the secure storage system and only the principal investigators will have access to the master list. Study statisticians and investigators will only have access to anonymized research data.
Process Evaluation
The success of clinical decision support relies on an uptake of the tools into practice. To optimize uptake, it is critical to determine how the tools are being used by the patient-pulmonologist dyad. A sampled subgroup of patients (n = 20) and all pulmonologists (n = 24) will be invited for the qualitative process evaluation aiming to further improve risk communication and clinical workflow. Patients will be selected based their BMQ-COPD scores to ensure maximum variation in representation. After the patients complete a brief demographic questionnaire, we will conduct one hour, semi-structured, telephone interviews to explore differences in patient experiences, assist with interpretation of trial findings, and provide direction for any further refinements to the intervention.
To gain insight into pulmonologist’s satisfaction, we will use Normalization Process Theory (NPT) to better understand the factors that affected their implementation, with particular attention to clinical burnout and alert fatigue (40). The 16-item NPT questionnaire covers questions in four important domains: sense-making, participation, action, and monitoring (40). The information gained will be fed back to the design of the tools before largescale implementation.
Data Analyses
Sample Size Calculations
Details of the sample size calculations are provided in the Supplementary Material – Section 2. In summary, the sample size is based on a formula for a stepped-wedged cluster RCT under a cross-sectional sampling method (41, 42), accounting for the variance in inflation due to the intra-cluster correlation and an adjustment for unequal cluster sizes. The unadjusted sample size was based on a type I error of 0.05 and a power of 80%. The current rate of prescription appropriateness was based on our analysis of BC’s administrative health data (43). The unadjusted sample size was estimated to be 528 per arm.
The design effect is a function of the overall design features (i.e., the cross-over feature, number of clusters, run-in period), as well as the intra-cluster coefficients and cluster auto-correlation. These were obtained from a dedicated analysis of administrative health data from BC using prescription patterns from all physicians in the province (43). The estimated intra-cluster coefficients and cluster auto-correlation were 0.026 and 0.491, respectively, resulting in a design effect of 2.15. Finally, the design effect was further adjusted for unequal equal cluster sizes, giving rise to a value of 2.16. Based on these, the total sample size is 528 x 2.16 = 1,144.
To test the feasibility of achieving this sample, we performed an audit of the 2019-20 fiscal year in one of the study sites, and recorded 5,559 visits. We estimate 17% of visits are due to COPD and eligible for this study. Given patient traffic in this site is expected to contribute to 60% of the overall traffic, and the two-year recruitment window, the expected total number of eligible patients will be 3,024. Therefore, to achieve the desired sample size, we will need a participation rate of 38% in the cross-sectional component of this study, a figure that we consider achievable given our experience with clinical studies at both sites and that the cross-sectional component of this study consists of a short 30-minute post-visit interview.
Statistical Analyses
Following an intention to treat principle, clusters will be analyzed according to their randomized crossover time regardless of whether crossover was achieved at the desired time. We will use a generalized linear mixed model (GLMM) in our analysis. This approach employs a shared random effect term to account for multiple correlated visits for each pulmonologist (cluster), as well as other available covariates to adjust for potential confounders. Given the binary nature of the primary endpoint (prescription appropriateness), we will use a logistic GLMM. For our secondary outcomes, we will examine different standard continuous distributions in the GLMM to find the best fit to the data (according to the major goodness-of-fit measurement such as Akaike information criteria [AIC] and prediction error). Additionally, time will be added as a fixed effect to capture trends in the outcome over the course of the study (steps) (41, 44). To investigate whether the intervention requires an adjustment period (before becoming fully embedded), we will examine the length of exposed time for each pulmonologist as a fixed effect modifier. Lastly, we will use multiple imputation methods for all missing data that can be considered missing at random.
To test the sex- and gender-related hypotheses, we will conduct a principal component analysis in which gender-sensitive variables (Supplementary Material – Section 1) are combined into a scalar ‘gender index’ (45). To examine the effect of this index on the accuracy of risk prediction, we will use established statistical methods to examine the incremental value of a new predictor (improvement in c-statistic, bias- corrected net reclassification index, and decision curve analysis) (46).
For qualitative data analysis, interviews will be audio-recorded and transcribed, in addition to detailed note-taking throughout the interviews. The transcripts will be uploaded to NVivo 12 for analysis. The interviews will be analyzed using the constant comparative method (47) to determine differences and similarities within and across participants. Two researchers will iteratively engage in coding, categorization, and refinement of the datasets until a cohesive thematic framework is achieved.