We aim to conduct a pilot implementation study with the following aims:
- Estimate the effect of an Opt-Out versus Opt-In Tobacco Treatment System on the proportion of mental health patients who are treated for tobacco use by their psychiatrist.
- Assess provider fidelity to the Opt-Out System, provider perceptions of the Opt-Out System, and barriers and facilitators to implementation of the Opt-Out System.
- Estimate the effect of the Opt-Out versus Opt-In Tobacco Treatment System on use of cessation treatment and abstinence among mental health patients who smoke.
Figure 1 displays an overview of our methods and study design. We will use a mixed-methods, two-arm cluster-randomized study design. Because the study is targeting provider behavior change, we will randomize at the provider level.
This study will take place at the VA New York Harbor Healthcare System (NYHHS), which serves primarily low income Veterans in New York City and surrounding areas. Approximately 60% of NYHHS patients are Caucasian, 31% are African American, 17% are Hispanic, and 61% have a high school education or less. The VA NYHHS receives approximately 14,000 mental health clinic visits each year, and 40% of mental health patients have documentation in the electronic health record (EHR) of current smoking. The VA NYHHS has pharmacotherapy and behavioral counseling available for all smokers.
The overall study approach is guided by Proctor’s framework for implementation research and the Consolidated Framework for Implementation Research (CFIR).20 The Proctor framework includes three main processes to implementation research: 1) the selection of an evidence-based practice (EBP), 2) the development of strategies to implement the EBP, and 3) outcome measurement, which includes implementation outcomes, service outcomes, and client/patient outcomes. The CFIR identifies five main domains that can influence implementation outcomes: 1) an organization’s inner setting, 2) an organization’s outer setting, 3) the characteristics of the intervention, 4) the perceptions and characteristics of individuals who interact with the intervention, and 5) the implementation process.
Figure 2 displays our study approach within these frameworks. The study is designed to improve psychiatrist adherence to an evidence-based 3A’s model of tobacco treatment – asking about use, advising to quit and assisting by prescribing NRT and referring to counseling. We will test two strategies for increasing psychiatrist adherence to this approach that will combine provider training with EHR system change to drive provider behavior. During Aim 2, we will measure implementation outcomes such as fidelity, acceptability, acceptability and perceived sustainability of the implementation interventions, as moderated by CFIR domains. During Aim 1, we will measure the effectiveness of the implementation interventions on provider perceptions of the Opt-Out System, and barriers and facilitators to implementation of the Opt-Out System. During Aim 3, we will measure the impact of the implementation interventions on patient use of treatment and abstinence.
Implementation Strategy Framework
Figure 3 displays the theoretical framework that guided the design of the study’s implementation interventions, which combines the psychological Theory of Planned Behavior and the Default Bias Theory from the field of behavioral economics.21 The Theory of Planned Behavior (TPB; shaded boxes in Figure 2) posits that behavioral intentions are the most proximal determinant of behavior, and there are three primary antecedents of intentions: 1) attitudes/beliefs about a behavior, 2) perceived social norms about the behavior, and 3) perceived control over performing the behavior. The TPB also suggests that one’s actual behavioral control influences perceived control and directly influences behavior. The TBP has been shown to predict health care provider behavior in prior research,22 and the literature and our prior work have identified tobacco treatment barriers within psychiatry that align with the constructs of the TBP. For example, psychiatrists report that lack of training (knowledge) leads to low self-efficacy and comfort (perceived behavioral control) in treating tobacco and in working with patients who are resistant to treatment, and that limited time (actual behavioral control) and lack of prioritization of tobacco among leaders and peers (social norms) limits their provision of tobacco treatment.10 Provider training programs and traditional (opt-in) clinical reminder systems are designed to overcome such barriers that align with the TBP.
However, behavioral economic theory posits that cognitive biases (black box in Figure 3) lead directly to human choice and behavior, independent of one’s beliefs or intentions about a behavior.13,14 One such bias is the default bias (or status quo bias), which causes humans to select customary or default options even in the presence of superior or healthier alternatives. Opt-out systems take advantage of these cognitive biases by making the desired behavior (in our case, tobacco screening and treatment) the default.
Table 1 outlines the components of the study’s implementation strategies and the constructs from the guiding theoretical framework (Figure 3) that the components are designed to target.
Both Arms: Psychiatrist Training and Academic Detailing
Since psychiatrists cite lack of training as barrier to providing smoking cessation treatment, both arms will receive a 1-hour training in helping their patients quit smoking.23 The training will be adapted from the evidence-based Psychiatry Rx for Change training program for psychiatry residents.24 We will also include topics that emerged as treatment barriers or facilitators during our previous qualitative interviews with psychiatrists, such as linking cessation to improved mental health treatment outcomes. To reinforce provider knowledge and self-efficacy in treating mental health patients for smoking, we will also implement academic detailing for providers in both arms. Academic detailing has consistently shown improvements in provider behavior.25 Two study investigators will make a brief outreach visit to each psychiatrist, at least one of whom will be a physician to answer any questions about cessation medications. The investigators will follow the seven steps recommended by the National Resource Center for Academic Detailing:26 1) The Introduction; 2) Needs Assessment; 3) Key Messages/Features/Benefits; 4) Understanding Barriers and Enablers; 5) Identifying and Handling Objections; 6) Summary; and 7) Close. The key message (Step 3) will include demonstration of the clinical reminder, review of the evidence for smoking cessation medications and how to prescribe them, and review of the role of the facility Health Promotion staff and how to contact them. Detailing will occur after randomization, so each psychiatrist will receive a detailing visit on the specific intervention to which he or she will be exposed.
Arm 1: OPT-IN Clinical Reminder
To increase actual and perceived behavior control and to increase perceived prioritization (disciplinary norms) of the treatment of psychiatric patients who smoke, we will implement a tobacco clinical reminder embedded within the EHR. Tobacco use clinical reminders are a best practice recommended by the PHS tobacco treatment guidelines and are routinely tested and used to address tobacco in primary care settings.5 All VA facilities currently use clinical reminders, which are adapted locally and can be adapted for specific individuals or groups of providers. The reminder will guide providers through each step of the 3A’s approach:
- Ask and Advise – Providers will be prompted to ask their patients if they currently use tobacco and advise the patient to quit and to use treatment if interested in quitting.
- Assist: Medications – Providers will be able to order cessation medications by clicking a box associated with an order template embedded in the reminder. The VA already has pre-set ordering templates for NRT (patch, gum, and lozenge), bupropion, and varenicline.
- Assist: Referral to Counseling – Providers will be able to refer patients to the local cessation counseling program by clicking a consult box embedded in the reminder. The consult will be sent to the facility’s local cessation program.
Arm 2: OPT-OUT Clinical Reminder
For Arm 2 we will directly change the treatment status quo by implementing a clinical reminder that automatically initiates an order for NRT and referral to the cessation program at the time a smoker is identified. The psychiatrist will need to actively cancel the NRT and counseling orders in order to opt the patient out of treatment. The reminder will include the following domains:
- Ask and Advise – Providers will be prompted to ask their patients if they currently use tobacco. Psychiatrists will be prompted to advise patients that the VA’s goal is to help all patients quit by prescribing NRT and referring them to tobacco cessation coaching.
- Assist: Automatic Medications – The VA has pre-set ordering templates for NRT, bupropion and varenicline. For smoking patients, the psychiatrist will receive an alert that an order for combination NRT (patch plus gum) will be placed unless the provider cancels the order by clicking a box within the reminder.
- Assist: Automatic Referral to Counseling – For smoking patients, the reminder will also automatically generate an electronic consult to the local smoking cessation program (described above). The reminder will include a box to check if the psychiatrist does not want the Coordinator to follow up with the patient.
One month before implementation of the provider training and reminders, all psychiatrists in the facility will be notified of the study at a required staff meeting. Psychiatrists will have the option at the meeting to ask questions and provide written or verbal opt-out of study participation. Psychiatrists will also have the opportunity to ask questions one-on-one with study investigators before deciding whether to participate. Psychiatrist turn-over is low at the facility; however, during recruitment we will ask psychiatrists if they plan to leave in the next 6 months and only enroll those who have no plans to leave. We anticipate that 20 of 24 psychiatrists at the VA NYHHS will participate.
Psychiatrists who do not opt-out of participation will be randomized to one of the two study arms, stratified by site, supervised by the study’s statistician. Psychiatrists who opt-out of study participation will receive the opt-in clinical reminder as a part of their routine care, but their performance will not be included in study analyses.
The study’s primary outcomes are: 1) the percent of smokers prescribed a cessation medication and 2) the percent of smokers referred to cessation counseling. Secondary outcomes will include patient use of cessation treatment and self-reported 7-day abstinence at 6 months. We will also measure intervention fidelity and provider perceptions of the intervention components.
Data Sources and Measures
Aim 1: Cessation Prescriptions and Treatment Referrals
Table 2 outlines our measures, data sources and data collection schedule for the study. Our assessment plan for Aim 1 is to use VA administrative data to estimate and compare the effect of the Opt-Out versus Opt-In treatment systems on the percent of all smokers treated for tobacco use by their psychiatrist during the study’s intervention period. The VA uses a fully EHR system that documents diagnostic and procedural data from all outpatient and inpatient encounters. The VA’s Informatics and Computing Infrastructure (VINCI) allows VA-affiliated researchers to query encounter data and has data analysts available to assist investigators with data selection. We will work with VINCI programmers to identify all patients seen by a participating psychiatrist in the six months before and after implementation of the clinical reminder. We will then ask VINCI programmers to calculate the percent of these patients who were screened for tobacco use by the psychiatrist, and then among smoking patients, the percent prescribed at least one cessation medication and the percent who were referred to the local cessation program.
Aim 2: Fidelity, Provider Perceptions, and Implementation Barriers and Facilitators
Training Logs: We will log all training activities to capture the proportion of participating psychiatrists who attended the training sessions and received an academic detailing visit, and the content and length of the academic detailing visits.
Post-Visit Surveys: Once a month during the 6-month intervention period, research assistants blinded to group assignment will survey a random sample of patients seen by a participating psychiatrist within 24 hours of their visit to assess provider fidelity to the 3 A’s approach. To identify and recruit patients for the surveys, we will use the EHR to identify a list of patients seen by a participating psychiatrist on the day we run the EHR query. We will take a random selection of 5-10 male patients (depending on response rate) and all female patients (to increase representativeness of women) per psychiatrist at each monthly assessment point to reach out to for a survey. We will make 2 attempts over 24 hours to reach the patient by phone to explain the study and obtain verbal consent. We aim to complete post-visit surveys with 20 patients per psychiatrist over the intervention period (N=200/arm). Participants will receive $10 for completing the survey.
Provider Perceptions and Implementation Barriers/Facilitators
Observations: During the group training sessions, the study coordinator will take notes and the trainers will complete reflection memos after each session that capture the psychiatrists’ reactions to the training content, questions asked and any group discussion. The coordinator will also observe each academic detailing session and document psychiatrist reactions, comments and questions. All study meeting minutes (excluding confidential information) will be analyzed qualitatively for themes related to provider perceptions and barriers/facilitators.
Provider Survey: We will use a repeated measures design to conduct a survey with participating psychiatrists at baseline and 6 months assessing their attitudes, beliefs, motivations and intentions to treat tobacco. We will invite providers to participate in the survey by sending an email to their VA email address. This email will include all elements of informed consent and a link to complete an online survey through the VA’s secure REDCap system.27 We will send two reminders to non-respondents. Providers will be paid $10 for the completion of each survey. The survey will assess: attitudes toward the opt-in or opt-out reminder using the Evidence-Based Practice Attitude Scale (EBPAS),28,29 perceived level of control, subjective disciplinary norms and intentions to help their patients quit smoking using questions adapted from the Determinants of Implementation Behavior Questionnaire (DIBQ),30 and intrinsic and extrinsic motivations to treat their patients for tobacco using items from the Treatment Self-regulation Questionnaire31
Provider Interview: Guided by the Proctor and CFIR models, we will conduct semi-structured interviews with 12-14 psychiatrists (6-7 per arm) assessing their views on the appropriateness, acceptability and sustainability of the intervention components, as well as how the clinic’s inner setting (e.g., culture, norms, workflow compatibility with the intervention), outer setting (e.g., VA policies, psychiatric professional associations), psychiatrist characteristics (e.g., beliefs about the intervention), and the implementation process (e.g., how providers were informed, who championed the intervention) may impact their views on appropriateness, acceptability and sustainability. We will also ask psychiatrists for their insights into challenges and successes encountered during their participation in the intervention components. We will recruit psychiatrists for interviews using sign-up sheets and by institutional email invitations. A trained interviewer will follow an interview guide with a set of pre-specified questions and follow-up probes. All interviews will be audio-taped. Psychiatrists will be paid $20 for completing an interview.
Aim 3: Patient Use of Cessation Treatment and Self-Reported Abstinence
We will assess patient use of cessation treatment and tobacco abstinence 6 months after seeing a participating psychiatrist. For this aim, we will conduct a follow-up telephone survey with the 400 cluster-sampled patients who completed a post-visit survey during the intervention period. The follow-up survey will ask patients to indicate whether they used a list of tobacco treatments in the prior 6 months, including all FDA-approved cessation medications, in-person cessation counseling, telephone cessation counseling, and a mobile texting cessation service. Consistent with guidelines for measuring abstinence in pragmatic trials, the survey will also assess 7-day point prevalence abstinence.32 We will make up to 10 attempts at different times of the day and month to reach patients by phone for a follow-up survey. Telephone non-respondents will be sent a survey in the mail with a pre-paid return envelope.
Other Measures: Patient Characteristics
Our patient surveys will collect additional information, including sociodemographics (age, gender, marital status, race/ethnicity, income), attitudes toward tobacco treatment using the Attitudes Toward Nicotine Replacement Therapy Scale33 adapted to ask about NRT and counseling, quitting self-efficacy using the Smoking Self-Efficacy Questionnaire,34 motivations to quit using a 0-10 scale, and smoking status and history using questions from the California Tobacco Survey. 35
We will first summarize survey and administrative data using descriptive statistics (means, medians, standard deviations, frequency distributions, and graphical displays) to characterize providers and patients treated by the providers in the two intervention arms.
Based on administrative data, we will categorize the proportion of smokers seen by a participating psychiatrist in each group who: (1) received a cessation medication prescription from the psychiatrist or (2) referred to cessation counseling by the psychiatrist. We will use generalized linear mixed effect models (GLMMs) with random effects for the clusters to compare these screening and treatment rates between groups. For all analyses, sensitivity analyses will be used to evaluate the impact on results from missing data and subject dropout. Specifically, missingness will be handled by creating a separate category, or removed by multiple imputation (MI).
For intervention fidelity, we will use descriptive statistics to summarize the patient post-visit survey data to calculate the proportion of patients seen by a participating psychiatrist in each study arm who were asked about smoking, offered medications, offered counseling referral, and (among those offered) the proportion of patients who accepted medications and referral. We will also summarize training fidelity logs (the proportion of psychiatrists who attended the trainings and received an academic detailing session).
To assess provider perceptions quantitatively, we will summarize provider survey data using descriptive statistics (means, standard deviations) to understand providers’ attitudes and beliefs regarding treating their patients for tobacco. For our qualitative data analysis (interviews, observations, meeting minutes), we will use a three-step coding process for each data source. First, two investigators will individually read a sub-sample of data (e.g., 3 interviews) to identify preliminary inductive codes, then meet to achieve consensus on coding the sub-sample and create the first draft of a code book. Second, the investigators will individually apply the codebook to a second sub-sample of data, and meet to achieve coding consensus on the second sub-sample to create final codebook for the data source. Third, once all data are coded, the investigators will meet to complete more focused coding to identify code clusters, relationships among codes, and common themes. Once all data sources are coded, we will also use group consensus meetings to look for themes across the main data sources.
We will use a similar analytic approach as in Aim 1. We will first categorize each patient as: (1) having achieved or not achieved 7-day abstinence 6 months after their psychiatry visit and (2) having used or not use any type of cessation treatment in the 6 months after their visit. This will estimate use of treatment and abstinence rates for each study arm. We will use GLMMs to compare these two outcomes between groups. Sensitivity analyses will be used to evaluate the impact on results from missing data and subject dropout. An ITT approach will be compared with the complete-case only method, but our primary analysis will be the complete case analysis, as The North American Quitline Consortium has found this approach to be more accurate in representing true quit rates and recommends use of this calculation,36 as do other reviews.37
Sample Size and Power
We aim to enroll all practicing psychiatrists at our study site, but have conservatively estimated that 20 will enroll. We calculated the level of power this will provide us to find a significant group effect on the proportion of patients prescribed a cessation medication by their psychiatrist (primary outcome) at all during the intervention period. Our power calculation varied the intraclass correlation (ICC) of 0.05-0.15 based on a cluster-randomized trial of preventive care in primary care practices.38 We estimate that 10% of patients in Arm 1 will receive a prescription.8 With the smallest ICC of 0.05, a type I error of 5%, and 80% power, 20 clusters (psychiatrists) will provide us with enough power to detect a 21% or greater prescription rate in Arm 2. With the largest ICC of 0.15, a type I error of 5%, and 80% power, 20 clusters will provide us with enough power to detect a 29% or greater prescription rate in Arm 2.