Design.
This program evaluation will use mixed (e.g., qualitative and quantitative) methods [18] to determine the impact of the ACP-GV National Program. As an observational implementation evaluation (see Fig. 1), we will start by using a propensity score (PS) matched control design and VHA administrative data to determine the proportion of ACP discussions in VHA by comparing ACP-GV sites to PS matched control sites not implementing ACP-GV (Aim 1). As a subgroup analysis, we will also determine whether individual strategies or combinations of strategies impact the outcomes of ACP discussion and AD completion rates across the VHA, with special attention to differences between current FY 2019 sites that do (n = 23) and do not have (n = 12) ORH funding (i.e., DEI-unfunded sites) (Aim 2). For Aim 3, we will conduct a budget impact analysis of the ACP-GV National Program. Finally, qualitative methods will be used to identify the characteristics of high-performing (e.g., high rates or sustainers) and innovative sites (e.g., unique local program design or implementation of ACP to inform sustainability and further spread). (Aim 4).
Sample.
The population for this evaluation potentially includes all Veterans who seek care in FY 2019 in VHA (N = 9.05 million). The total number of VHA facilities is approximately 170 sites. All Veterans who seek VHA services are eligible to receive information on ACP and can complete an AD. Some Veterans may choose to engage in an ACP discussion individually while others may elect to attend a group visit. While all Veterans are eligible, not all Veterans will engage in ACP discussions or ever complete an AD. Using VA administrative data, we will examine all documented ACP discussions, AD completions, and ACP-GV encounters to estimate the sample. In FY 2019, there were 23 funded sites and 12 unfunded sites (i.e., no funding for dedicated staff positions), for a grand total of N = 35 sites.
Data Sources.
There are three main data sources for this evaluation: (1) VHA administrative data including data obtained from the VA Corporate Data Warehouse, for data at the patient, provider, clinic, and facility levels; (2) ACP-GV Implementation Team Site Tracking Reports, monthly reports from sites on mandatory data elements, Diffusion of Excellence Initiative hub site-level entries, and national data collected on the sites’ participation from initial recruitment to full participation in all of the ACP-GV National Program implementation activities noted in Table 1; and (3) ACP-GV Evaluation Team qualitative interviews, which will collect data on budget impact and implementation consequences, characteristics, and innovations.
The VHA administrative data used here obtains data from the VA Corporate Data Warehouse which includes patient (e.g., name, social security number), provider (e.g., document definition, stop codes, visit date), site (e.g., site name and number), note title (e.g., AD note, AD discussion, AD rescinded, other AD note category) and an ACP-GV-specific workload code. The ACP-GV Implementation Team Site Tracking Report was created specifically to track and to manage the ORH-funded grant activities from recruitment to full-scale implementation as an ACP-GV site. This report tracks all implementation activities offered to any VHA facility that expresses interest in and/or later applies to become a funded or unfunded site. As noted earlier, ORH funding is awarded to rural sites, typically for a social worker and medical support assistant, with the Full Time Employee Equivalent (FTEE) level based on the site’s projected annual workload. The FTEE, amount of funding, and type of positions for staffing of each ACP-GV site is tracked by the ACP-GV Implementation Team in the Site Report.
Data Collection for Aim 1 and 2 and 2.
To complete Aims 1 and 2, we will use VHA administrative date from the VA Corporate Data Warehouse data to create patient, clinic, facility and site level variables, including Veteran demographics, service utilization, AD note titles and ACP-GV workload. For Aim 2, we will merge this data with site-level variables from the ACP-GV Implementation Team Site Tracking Reports, which include sites with funded FTEE and an estimate of the unfunded sites staffing patterns. We will then add to the above, data collected on the ACP-GV sites’ participation in the implementation activities from the ACP-GV Implementation Team’s Site Tracking Reports.
For Aim 3, we will develop protocols with the ACP-GV Implementation Team to collect new data using an adapted version of the Time Tracking Tool developed by the Behavioral Health Team Based QUERI [19], which will quantify and connect the amount of time each staff member spends in implementation activities to allow for analysis of each strategy alone or in combination. Because implementation costs will be obtained from sites at varying stages of implementation, we will define the index date of transition from pre-implementation to implementation for an individual site as the date of initial use of the ACP-GV specific workload code and AD discussion note by that site, while sustainability will be assessed using data for the 12 months following the index date when they started implementing ACP-GV. We will reconcile this time tracking data by allocating each cost to a site-specific period (i.e., pre-implementation, implementation, or maintenance) relative to the site-specific index date to use as implementation cost data for Aim 3.
Aim 1: Evaluate the impact of the ACP-GV National Program on the proportion of ACP discussions in VHA by comparing ACP-GV sites to propensity score matched control sites not implementing ACP-GV.
Design.
The 35 ACP-GV sites will be considered “intervention” sites. We will identify a set of matched “control” sites (N = 35) defined as VHA facilities not implementing ACP-GV based on an analysis of the ACP-GV workload specific code in VA Corporate Data Warehouse data and not participating in the National Program implementation activities but documenting ACP discussions as noted in the VA Corporate Data Warehouse and using AD note titles. Our hypothesis is that ACP-GV sites (i.e., intervention sites) will have a higher proportion and significantly greater quarterly and annual rates of ACP discussions compared to the PS matched control sites. Given that the sites were not randomized to either the intervention or control, we will use PS matching to create a “pseudo” randomized evaluation design. The PS is a model-based conditional probability of a site’s membership in the intervention, given its observed site characteristics [20]. We will use a logit model to estimate the PS for each site. The dependent variable of the logistic regression model is being/not being in the ACP-GV intervention group. The independent variables are site characteristics including: rurality, geographic location, Veterans Integrated Service Network (VISN), staffing, facility size, and aggregate Veteran demographics such as age, gender, and race/ethnicity. We will use a greedy matching algorithm based on 5-to-1 digit matching [21]. An initial ACP-GV site will be randomly selected from the existing sites to start the process. A non-ACP-GV site (i.e., control) with the closest PS that lies within a fixed distance will be selected for matching. If multiple control sites have PS that are equally close, then one of the control sites will be selected at random. The process will continue until all ACP-GV sites that can be matched have been matched. To evaluate the success of the PS matching, we will compare the balance in the distributions of all the observed site covariates between the two groups using standardized difference plots. An absolute standardized difference of less than 10% suggests a negligible imbalance between the two groups for a given site covariate [22]. A PS matching limitation is if no control sites have PS that lie within the predefined distance (i.e., caliper) of an intervention site, then that ACP-GV site is not included in the PS matched sample. Given sample size limits, if the PS matching produces a limited number of pair matches, we will consider other strategies, (e.g., PS through stratification, covariate adjustment with PS, or inverse PS weighting).
Data Analysis for Aim 1.
We will aggregate patient-level outcomes of the ACP discussions and site characteristics (e.g., geographic location, facility size, rurality, etc.) across VHA sites quarterly and annually for descriptive purposes, and then compare the proportion of ACP discussions with the PS matched control sites. We will specifically examine sites with a high annual rate of ACP discussion and low ACP-GV rates and vice versa to identify patterns and trends. We will present numerical variables as means (SD) or medians (interquartile-range), and categorical variables as counts and percentages (with 95% confidence interval). We will use a significance level of 0.05. To compare annual rates of ACP discussions, the primary outcome for this Aim, we will use a two-sided z-test with pooled variance to test the difference in proportions across the two groups (i.e., ACP-GV vs. non-ACP-GV). As a secondary analysis, multivariable Poisson regression will examine the differences in annual rates of ACP discussions, while accounting for potential imbalance in site characteristics. Given the sample size, we will build a parsimonious model based on forward model selection.
Aim 2: Among ACP-GV sites, document and compare ORH-funded and DEI-unfunded sites on the effectiveness of implementation strategies (individual and combinations) used by sites in the ACP-GV National Program on ACP discussion and AD completion rates across the VHA.
Data Analysis for Aim 2.
We will use a comparative case study approach to compare funded sites to unfunded sites on the two outcomes of ACP discussion and AD completion rates for Aim 2. Due to the small number of sites, we are hindered in our ability to control for confounders. Yet, we anticipate that additional funding for staff positions will largely impact outcomes. Therefore, we will track the other four implementation strategies (see Table 1) to serve as mediators of impact. If funding expresses greater participation in the other strategies, then there may be a mechanism of action within accessing new funding alone, or because it is in combination with the other strategies.
In our preliminary data, there is some evidence of variation in participation in the implementation strategy of ongoing training by funded sites and increased uptake of ACP-GV vs. unfunded sites. We will use the Implementation Team Site Tracking Report and qualitative interviews from Aim 3 and Aim 4 to determine if we can make quantitative inferences as to what funding imbues and what impact it has on the other implementation strategies. To provide feedback to our partners for quality improvement and to not lose important trends that may exist in the data that may help understand larger trajectories of change, we will generate quarterly reports tabulating the number and type of implementation strategies used at each site per quarter and the extent to which each of the sites achieved the implementation outcomes. The annual summative evaluation report will describe the implementation strategies that were used by multiple facilities which were most successful and least successful in achieving positive outcomes. These quarterly briefs, annual reports, and our Aim 4 analyses will be synthesized to support sustainability and spread of the initiative.
Aim 3: Determine the budget impact of the ACP-GV National Program.
Design.
To accomplish Aim 3, we will follow the steps for conducting a budget impact analysis (BIA) [23] developed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) [24], and referenced by Smith and Barnett in their discussion of the role of economics in the QUERI program [25]. Implementation- and program-level costs will be collected and analyzed to account for the full impact of the ACP-GV program at individual sites and nationally for VHA.
Data Sources for Aim 3.
Data will be collected for the pre-implementation and implementation periods to estimate costs associated with ACP. VHA Managerial Cost Accounting (MCA) System data will be used to estimate health care encounter and VHA payroll costs. MCA is a derived database built from standard VHA data sources that allows comparison of cost and utilization characteristics. Activities associated with implementation (implementation period only) and direct provision of the ACP-GV will be collected at the local level and aggregated for the national program. Productivity and time data will be collected using a combination of the VA Corporate Data Warehouse, the Time Tracking Tool, and stakeholder interviews conducted as part of Aim 4. Costs will be assigned to these activities based on VHA payroll costs in MCA.
Data Analysis for Aim 3.
ACP-GV is not expected to have direct short-term impact on overall patient healthcare utilization; therefore, the focus of the analysis will be costs related to implementation and provision of ACP-GV for sites and the National Program. Changes in costs are likely to vary by site depending on the extent to which existing resources are leveraged for provision of ACP-GV. The ability to provide ACP discussions to multiple Veterans in a single session via ACP-GV will offset these increased costs; however, the extent to which these two factors exist will vary by site. To estimate direct costs, the time associated with all implementation and program activities will be multiplied by the salary and fringe rate of the team member performing the activity. Each activity will be assigned to a specific time, site, and team member to allow for estimation of costs at the individual site level. Indirect implementation costs (e.g., educational materials) will be tracked and allocated to sites by the ACP-GV Implementation Team.
To examine the budget impact at existing sites, we will compare the costs associated with ACP during the year before and after the site joined the ACP-GV National Program. Cost differences will be analyzed from the payer perspective [23]. For the national program, including all ACP-GV program costs and costs attributed to the implementation strategies identified in Table 1, VHA will be the payer. For program and implementation costs specific to a VAMC, the VA Medical Center (VAMC) budget administrator will be considered the payer. Observable characteristics that are predicted to influence the ACP budget of an individual site will be collected (e.g., VAMC size). Using TreeAge software, a probabilistic sensitivity analysis (PSA) will be conducted [26]. The PSA allows the assignment of a distribution to model parameters rather than just a point estimate to account for uncertainty in the BIA model. To facilitate broader implementation of ACP-GV, a decision analysis model is needed for non-participating sites to estimate the budget impact at their site based on site-specific factors (e.g., VAMC size). The model will be developed in Microsoft Excel and a user guide will be created and disseminated to sites. This model will show the budget impact at existing ACP-GV sites as well. One-way sensitivity analyses will be possible within the model to allow decision-makers to see the impact of each site-specific factor.
Aim 4: Identify the characteristics of high-performing (e.g., high rates or sustainers) and innovative sites (e.g., unique local program design or implementation of ACP) to inform sustainability and further spread.
Sampling Plan.
We will purposively select approximately 24 ACP sites. We will use site reports and our Aim 1 and 2 VA Corporate Data Warehouse datasets to identify a total of six sites with high-performing characteristics (i.e., three sites with high productivity in terms of ACP-GV against the performance goal of 200 ACP-GV groups a year/per FTEE and three sites with high rates of AD discussions in the 75th or higher percentile nationally) and six sites that are sustainers after the intensity of implementation strategies begins to reduce (i.e., three sites that meet the ACP-GV performance goal and three sites with high rates of AD discussions). To capture unique site-specific program design or other innovations that might be missed by identifying sites based only on performance, we will identify six sites based on unique local program design characteristics and six with innovative models for implementing ACP known to the ACP-GV Leadership or Implementation Team or from data in the ACP-GV Implementation Team Site Tracking Report.
Data Collection for Aim 4.
Currently, the ACP-GV National Program and Implementation Team host weekly educational calls for all sites. The Implementation Team routinely tracks training participation at the individual and site level and records it on the ACP-GV Implementation Team Site Tracking Report. Prior to starting Aim 4 interviews, we will recruit sites on these calls by describing the opportunity for ACP-GV site personnel who are implementing the program to participate in interviews. We will conduct semi-structured qualitative interviews with the main implementer of ACP-GV at each selected site. These primary points of contact are tracked on the Site Tracking Report as participating in the ACP-GV National Program’s activities. Given that our operational partners deemed this evaluation to be quality improvement and non-research, the ACP-GV Implementation Team will send an e-mail, a copy of an information sheet, and the interview guide to invite the main implementer at the ACP-GV sites to participate. After confirming willingness to participate, a trained qualitative interviewer will conduct the 45-minute interview by telephone. To speed the process of transcription, a trained data entry assistant will digitally record this interview using a VHA approved recorder. Interviews will be transcribed verbatim by the data entry assistant (who will also be on the call) and checked against their notes. The interviewer will then prepare an interview report and provide it back to the interviewee for member checking. In order to attain targeted recruitment numbers, the interviewer will work closely together with the ACP-GV Implementation Team in weekly meetings to determine which sites qualify in each category and to discuss the sites’ level of participation. If participation is low, we will ask the ACP-GV National Program to encourage participation verbally on the calls and in their email correspondence with sites.
Data Sources for Aim 4.
The interview guide will be developed using the Consolidated Framework for Implementation Research (CFIR) Interview Guide Tool and includes the following constructs related to implementation, sustainability and spread: (1) Intervention Characteristics: Adaptability, Cost; (2) Outer Setting: External Policies & Incentives; (3) Inner Setting: Networks and Communications, Relative Priority, Organizational Incentives and Rewards; (4) Readiness for Implementation: Leadership Engagement, Available Resources; (5) Process: Planning; (6) Engaging: Opinion Leaders, Reflecting and Evaluating. Additional questions to be developed with input from the ACP-GV Leadership and Implementation Teams will focus on ACP-GV specific information on implementation and sustainability, as well as unique characteristics and innovations. Additional questions about the individual and set of implementation activities used, those not used, and rationales for each will be informed by the ACP-GV monthly site reports.
Data Analyses for Aim 4.
The coding team will include a doctoral trained qualitative interviewer/analyst, a data analyst, and a data entry assistant. We have divided the qualitative data analysis process into four steps: (1) Data management: Transcripts of the audio-recordings will be entered in Atlas-ti, a software program already on hand that enables analysts to mark blocks of text with codes. Coding will use systematic, iterative, and directed content analysis methods [27, 28]. Upon completion of independent coding by at least two coders, all the transcripts from multiple coders will be merged into one hermeneutic unit. (2) Development of Top-Level Codes: Following a deductive approach guided by domains identified from CFIR [29], we will use a two-tiered coding strategy (e.g., top- and sub-level). The team will follow the conceptual framework of the CFIR model and use the interview guide to develop a preliminary code book. (3) Top-Level Coding: The coding team will independently assign top-level codes to one-half of the transcripts using the preliminary code book, paying attention to themes that relate to performance and sustainability. To ensure coding reliability and consistency, the lead evaluator on the project will review initial and all final coding.
We will also use Coding Analysis Toolkit (CAT) software to run comparative statistics estimating team consensus in applying top-level codes. If the results do not meet the acceptable reliability criterion of 0.70 or higher, the team will meet to resolve differences. This process will be repeated until coding consistency at or above the acceptability level is achieved, after which the three coders will work independently. (4) Sub-level coding: The team will sub-code top-level codes that are “grounded” (i.e., associated with many quotations). Sub-level coding will follow a similar trajectory to that of top-level coding. After coding is complete, the team will select quotations that illuminate sustainability, innovation and uniqueness, and write summary statements and outline overarching themes across questions and codes, noting any repeating patterns or divergent ideas. Interpretation of the data will compare, contrast, and develop these overarching themes and prepare the summarized data into a final report highlighting variation to include unintended consequences of ACP-GV implementation and sustainability, as well as high-performing and innovative models of ACP. Lessons learned will be drafted from the strengths and limitations of the ACP-GV program. To integrate the quantitative and qualitative data, we will merge data from Aims 2 and 4 to refine the existing ACP-GV implementation manual to include sustainability and spread. We will refine the manual in concert with the ACP-GV Implementation Team.
Trial status.
The Institutional Review Board at Central Arkansas Veterans Healthcare System has approved this program evaluation. Data collection for this program evaluation began in October 2019.