An organization’s sustainability capacity (SC) represents its ability to implement and maintain the benefits of a systems change over time. Sustainability frameworks suggest that an organizations’ capacity to sustain change is influenced by multilevel factors or constructs related to organizational attributes, environmental contextual features, and intervention characteristics[1–4]. Examples of organizational attributes include leadership support, champion roles, revised policies and procedures or expert coaching support; external contextual features relate to regulatory or financial changes; and innovation attributes focus on ease of use or understanding how likely the benefits of the change would be sustained. Multiple studies have shown that various organizational, external, and innovation attributes inform the likelihood of sustaining an evidence-based practice within an organization[5–13].
Recent efforts have sought to define and refine the classification of these sustainability factors or constructs. The Integrated Sustainability Framework (ISF) identified 36 factors across multiple settings (e.g., community, school, clinical/social services) as being associated with sustainability[14]. These factors were grouped into five contexts: (a) outer context, (b) inner context, (c) intervention characteristics, (d) processes, and (e) implementer and population characteristics. Common factors within each context include sociopolitical context and funding environment (outer context); funding/resources, staffing and turnover (inner context); adaptability, fit with population or context and benefits/need (intervention); partnership/engagement and program evaluation (process); and implementer motivation and attitudes (implementer/population)[14].
Alternatively, the Consolidated Framework for Sustainability Constructs (CFSC) conceptualized 40 constructs across six themes associated with sustainability of change in healthcare settings[15]. The themes include: (a) initiative design and delivery, (b) negotiations related to the initiative processes, (c) organizational setting, (d) people or individuals involved, (e) resources, and (f) external environment. The CFSC also explored approaches (retrospective versus prospective) for assessing perceptions about sustainability and the level of focus (organizational versus intervention) associated with the assessment of sustainability capacity, which resulted in the identification of the ten most prevalent sustainability constructs within each category (Additional File 1)[15]. Four constructs are common across both level of focus and assessment timing, including demonstrating effectiveness and monitoring progress over time (initiative design and delivery), leadership and champions (people involved), and general resources to support sustainability (resources)[15]. However, other constructs varied according to whether they should be assessed at the organizational or intervention level. For example, training and capacity building, and integration with existing programs and policies were not typically assessed within an organizational level of focus. Other examples not assessed include staff perceptions about the belief in the initiative construct (intervention level of focus) and stakeholder participation (retrospective approach).
Despite extensive research on sustainability constructs and associated frameworks, few instruments have been developed for use in research to quantitatively assess staff perceptions about organizational sustainability capacity. Two such instruments are the Program Sustainability Assessment Tool (PSAT)[16] and the British National Health Services Sustainability Index (BNHS-SI)[17, 18].
The PSAT, a 40-item instrument, is organized across eight domains: Environmental Support, Funding Stability, Partnership, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, and Strategic Planning, and shows excellent internal consistency.[16] It is scored using a 7-point Likert scale (1 = little to no extent; 7 = a very great extent). The PSAT has been utilized to assess staff perceptions about sustainability for interventions targeting chronic conditions (e.g., asthma) and evidence based practices such as falls prevention strategies or alcohol screening and brief intervention [19–24].
The BNHS-SI predicts the likelihood of sustainability for an organizations’ improvement project[17]. The tool (see Additional File 2) consists of 10 factors designed to assess sustainability across three domains:
(1) Process– benefits beyond helping patients, credibility of the benefits, adaptability of the improved process, and effectiveness of systems to monitor progress;
(2) Staff– staff involvement and training to sustain the process, staff attitudes toward sustaining the change, senior leadership engagement, and clinical leadership engagement;
(3) Organization– fit with organization’s strategic aims and culture, and infrastructure for sustainability.
Respondents select one of four options for each of the 10 factors that best describes sustainability in their organization. The BNHS-SI utilizes an additive, multi-attribute, utility model to summarize the scores across the three domains which are then totaled to arrive at an overall organization sustainability propensity score[18]. The BNHS-SI has been utilized across multiple healthcare settings to assess staff perceptions about the sustainability of an organizational change [18, 25–35] and to qualitatively identify factors associated with the concept of sustainability[34, 36–38].
Constructs from the CFSC and the ISF represent elements often included in the structure of a quality improvement collaborative (QIC), such as understanding how implementation support and improvement methods might interact with stakeholder participation in training and capacity building activities to influence staff perceptions about an organization’s ability to sustain change. However, instruments such as the PSAT or the BNHS-SI have not been utilized to prospectively assess how staff perceptions about sustainability change over time while participating in a QIC.
NIATx200
The NIATx200 initiative built on prior successful NIATx research[39–43]. The objective of the NIATx200 initiative was to evaluate the effectiveness of implementation strategies commonly used a QIC. To achieve this objective, NIATx200 recruited 201 addiction treatment clinics in five states (Massachusetts, Michigan, New York, Oregon and Washington). Participating clinics were randomized to one of four implementation strategies: (1) interest circle calls (n = 49), (2) learning sessions (n = 54), (3) coaching (n = 50), or (4) a combination of all three implementation strategies (n = 48). An 18-month active implementation timeframe was followed by a 9-month short-term sustainment period. Structure of the NIATx200 initiative and the description of the implementation strategies are described in more detail elsewhere[44–46]. Primary outcomes consisted of wait time (mean days between first contact and first treatment), retention in treatment (percent of patients retained from first to fourth treatment session), and annual admissions. Organizational attributes such as non-profit status, size, or accreditation status were collected[47]. Secondary outcomes, which were collected at the staff level, included organizational readiness for change and perceptions about sustainability propensity.
Mixed-effect regression models determined which implementation strategy was most effective in improving outcomes, as well as being most cost-effective[44]. Improvements in the wait time and admission outcomes for clinics assigned to the coaching and combination strategies significantly differed from clinics assigned to the interest circle strategy[46]. For both wait time and admission improvement, the coaching strategy was more cost-effective as compared to interest circles[46]. No NIATx implementation strategy significantly improved treatment retention. Similarly, differences in attributes of the clinics (e.g., size) did not affect improvements in the outcomes.
Dissemination and implementation research has not, to date, explored staff perceptions about SC as an outcome measure, such as how staff perceptions about SC change over time and how participation in a QIC influences those changes. With this knowledge, implementation researchers might be able to modify implementation strategies or design sustainability interventions that target staff sustainability perceptions. Data collected during the NIATx200 initiative was utilized to begin addressing this empirical deficit.
Specific Aims
The specific aims of the current paper are to: (1) explore temporal changes in staff perceptions about sustainability and (2) assess how staff characteristics and organizational participation in a QIC influence changes in sustainability over time.