We chose a dynamic waitlisted design for the evaluation to accommodate WHS’s plans for staged EBQI implementation at 21 facilities over three years. This randomized “roll-out” implementation design has sound statistical properties, including higher power than traditional wait-listed designs (21) and less vulnerability to external, uncontrolled factors (20). See Table 1 for an overview of Evaluation Data Sources, Samples and Measures, described further below.
Data Sources and Measures
Key Stakeholder Interviews. Semi-structured qualitative interviews will be conducted at baseline for all sites and at 12-months post-EBQI launch, by group. Interviews will also be conducted with sites in Group 1 (the only group for which time permits a second follow-up) at 24 months. We will also interview WHS leaders and EBQI contractor personnel to evaluate leadership and implementation processes. Key stakeholder selection will be adapted based on the QI targets established at initial EBQI site visits (e.g., interview a mental health (mental health) director if the QI project targets mental health). We will seek to re-interview the same key stakeholders from baseline at follow-up but will pursue replacement personnel in the event of turnover and/or position changes over time.
The baseline interview guide includes questions about the structure and delivery of usual care for women Veterans, barriers and facilitators to achieving delivery of comprehensive women’s health care, what (if any) improvements are underway in women’s health and/or for women Veterans, familiarity with performance metrics, access to metrics by gender, experience with QI, local culture, perceptions of the care environment, and engagement of women Veterans in local initiatives (e.g., Women’s Health Council). The 12- and 24-month interviews will assess any changes in care for women Veterans (staffing, structure, etc.), details of completed/in progress QI projects, perspectives on critical components of EBQI, and anticipated sustainability of local improvements and QI methods. All key stakeholder interviews will be conducted by telephone, recorded and professionally transcribed. Transcripts will be reviewed and edited for accuracy.
Organizational Surveys. We will use key informant organizational surveys at baseline, 12- and 24-months among the 21 participating VA facilities, in addition to annual administered WATCH surveys from WHS. For Years 2 and 3, we will re-administer the same surveys, adapting selected domains in relation to EBQI targets of participating VAs. We will include measures of leadership support (22), local resources (e.g., sufficiency of time, personnel, equipment) (23), practice structure (e.g., women’s health care model, staff mix, referral arrangements), service availability (24), care coordination arrangements (within and outside VA), ability to engage in QI (e.g., barriers to QI, data access by gender), gender-sensitivity of environment (e.g., privacy), local challenges (e.g., provider shortages, hiring difficulties, practice chaos) (25, 26), facility type (e.g., size, academic affiliation, urban/rural), and EBQI activities (17). We will field surveys through REDCap, a VA-approved web survey vendor.
VA Clinician/Staff Surveys. We will use web-based clinician/staff surveys at baseline, 12- and 24-months that include measures of EBQI exposure/participation (e.g., awareness, hours spent, local buy-in), QI orientation/culture (e.g., perceived cooperation among managers/providers/staff, communication effectiveness, culture fostering flexibility, participative decision-making) (27–29), gender sensitivity (e.g., awareness, knowledge, attitudes, self-assessment of women’s health proficiency) (30), practice context (e.g., leadership norms, organizational readiness to change, job satisfaction, burnout) (31–33), and provider/staff characteristics (e.g., age, gender, race, ethnicity, staff type, clinician type, designated women’s health provider, proportion of women Veterans in panel/clinic, board certification, years in VA). We will obtain lists of local clinicians and staff by drawing a census from Primary Care Management Module data for each participating facility.
VA Administrative Data. We will pull secondary data on VA quality of care and patient experience, in addition to utilization patterns and other administrative data on women Veterans relevant to the evaluation. Measures will include process measures of quality for diabetes and cardiovascular disease (e.g., lipid screening) care and intermediate outcome measures (e.g., glycemic and lipid control), access, continuity, coordination, courtesy and overall satisfaction with VA care. Additional measures include access (e.g., average wait time, mental health), continuity (% of visits with PACT team providers), coordination of care (e.g., emergency room use), non-face-to-face access (e.g., telephone visits), utilization measures (e.g., outpatient women’s health, mental health, visit rates), and area measures (e.g., urban/rural location, academic affiliation, facility complexity score).
Qualitative Analyses (Aims #1 and #3). Analysis of key stakeholder interviews will initially focus on data consolidation (34) through the use of templated summaries (35) informed by the interview guide, and then organized into matrices to compare and contrast findings across roles, sites, and levels (e.g., facility, Veteran integrated service network (VISN)). In-depth analysis of the key stakeholder interviews will be done using ATLAS.ti, a qualitative data analysis software program that facilitates comparison of data across types and sources. Using a constant comparison analytic approach, the analysis team will develop a top-level codebook and refine it based on emergent themes, particularly as each round of interviews is completed (36, 37). Analysts will compare and contrast interviews within facility, across facilities, and over time. Consistent with our implementation-focused evaluation in the women’s health-PACT trial, we will explore which women’s health EBQI components are of particular value in improving care and examine clinic and provider characteristics associated with varying levels of EBQI effectiveness and achievement of comprehensive care.
Quantitative Analyses (Aim #2). We will examine multiple outcome measures as dependent variables: 1) multiple individual measures of comprehensive care achievement, including levels of women’s health service availability (as noted in VHA Handbook 1330.01) (14), integration of and access to gender-specific and mental health care, and other related measures that capture different domains of comprehensiveness; 2) gender-sensitive care delivery, including organizational and provider/staff level measures; and 3) quality of care and patient experience measures. For comprehensive care achievement, we will include as dependent variables the individual measures, and we will also examine approaches to creating an aggregated ordinal score of the individual measures. We will prioritize the final set of dependent variables in consultation with WHS.
The primary regressors of interest will be EBQI exposure (i.e., implementation) and time. We will examine the potential moderating effects of practice context and provider/staff knowledge/attitudes (e.g., determine EBQI effects in high vs. low leadership support sites). We will use multiple linear or logistic regression to evaluate EBQI effectiveness. Where appropriate we will adjust for covariates, account for clustering of patients by site, and mitigate bias due to non-response or loss to follow-up through the use of enrollment/attrition weights. Covariates used for adjustment will include patient factors (e.g., facility case mix, proportion of women Veterans seen), provider/staff factors (e.g., designated provider availability), and organizational factors (e.g., resource sufficiency, facility size).
Clustering by site will be accounted for by fitting hierarchical regression models with random intercepts for the sites using Stata 15 (38). We will evaluate the goodness-of-fit of a given regression model using standard diagnostics (e.g., Mallow’s statistic (Cp)) (39). To adjust for potential non-response bias and loss to follow-up over time for the provider/staff survey samples, we will apply enrollment weights using available characteristics of eligible providers/staff and attrition or “inverse probability of inclusion” weights estimated using an appropriately specified logistic regression model (40). We will use multiple imputation methods to replace missing values among covariates (41), with hot-deck methods used for imputation as needed (42). We will estimate site-level effects using the hierarchical regression models with random intercepts for sites. While our sample of sites (21) is small for the estimation site-level effects, EBQI trials of fewer sites have noted significant effects (43).