Sustainment is considered to be the final stage of the process of implementation of evidence-based practices, policies and programs (EBPs) [1, 2]. Defined as “the continued use of program components and activities for the continued achievement of desirable program and population outcomes,”[3] sustainment is considered to have occurred when, after a defined period of time, a program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behavior change (i.e., clinician, patient) is maintained, either as originally planned or with some degree of adaptation, while continuing to produce benefits for individuals/systems [4]. However, what is to be sustained differs from one program to the next [5, 6]. For instance, with respect to the community coalitions supporting drug and suicide prevention activities, some definitions of sustainability focus on the coalition itself while others focus on the activities and impacts of the coalition [7]. Moreover, sustainability is increasingly being viewed as a dynamic process with shifting outcomes that represents increased likelihood that the program or infrastructure will continue, while sustainment refers to the state of being sustained, often relating to the time after initial funding has ended [8, 9]. A major use of a sustainability measure is to provide feedback to grantee sites on their trajectory towards sustainment. The SMSS can be used for assessing both sustainability before initial funding ends as well as sustainment after this initial funding.
In recent years, there has been a proliferation of frameworks that focus specifically on sustainability [3, 4, 7-9]. However, despite the growing consensus as to how sustainability should be defined [3, 4, 9], the underdeveloped state of measurement of sustainment poses one of the most serious methodological challenges to understanding and facilitating sustainability of evidence-based practices and programs [9-11]. Some instruments like Stages of Implementation Completion (SIC) [12], have been developed to measure the sustainability outcomes of a specific intervention [13]. Other instruments like the Program Sustainability Assessment Tool [14, 15], The Program Sustainability Index [16] and the Sustained Implementation Support Scale [17]. take a broader ecological approach to sustainability and are used primarily to assess capacity and to plan for sustainability by measuring determinants or factors that influence sustainability. To our knowledge, there is no tool at the present time that assesses both sustainability determinants and sustainment outcomes.
One of the reasons for developing a valid and reliable measure of sustainment is to provide a means of conducting an audit and providing feedback to organizations engaged in implementing innovative and evidence-based programs and practices. While monitoring and feedback are recognized as important for prevention [18, 19], much of the relevant science on feedback in health has involved improvement in clinical performance [20-26]. This includes clinical supervision and use of technology like electronic dashboards in measurement-based quality improvement (MBQI) strategies that monitor patient behavior and clinician activity [27-29], while prevention has a more limited history of using computational technologies for monitoring [30-33]. Such feedback offers the clinician a better understanding of whether they are on course to achieve a successful outcome or need to alter their treatment in order to improve the likelihood of a successful outcome. MBQI strategies also hold great promise for facilitating implementation of evidence-based practices [34]. Audit and feedback has demonstrated that it can be an effective strategy for implementation [35, 36], but there is little evidence to suggest that it is equally effective in achieving the sustainment stage of implementation [37].
Federal agencies responsible for wide-scale delivery of prevention programs, including the Substance Abuse and Mental Health Services Administration (SAMHSA), routinely collect information from their grantees to monitor progress toward completion of goals and objectives. SAMHSA supports a wide array of prevention grant programs targeting mental, emotional, and behavioral disorders including substance abuse (SA), suicide, and antisocial behavior. Each of SAMHSA’s prevention initiatives have specific sets of goals and objectives, and each have different prevention approaches that the administration expects will be sustained once support from SAMHSA is no longer available. As part of their initial proposal for funding, all SAMHSA grantees are required to submit a plan for sustainment of the grantee’s activities once the federal funding has come to an end. SAMHSA programs currently rely on electronic data collection systems including the Transformation Accountability (TRAC) data collection system for SAMHSA’s Center for Mental Health Services (CMHS) programs, and the Coalition Online Management and Evaluation Tool (COMET) and the Performance Management Reporting Tool (PMRT) used by SAMHSA’s Center for Substance Abuse Prevention (CSAP). This information is used to provide feedback to grantees when there is evidence of failure to achieve goals and objectives. However, there is no empirical evidence that such feedback leads to an improvement in performance or increases the likelihood of sustainment.
While SAMHSA remains deeply concerned about sustainment, its mission does not allow monitoring of its grantees after funding has ended. In building a partnership with SAMHSA, the NIDA funded Center for Prevention Implementation Methodology (Ce-PIM, P30DA027828) responded to this need by undertaking the development of a sustainability measure across diverse prevention programs and test its predictive validity over time. This Sustainment Measurement System (SMS) was created as a tool that combined existing sources of information for SAMHSA grantees with information gathered by an instrument that assessed determinants as well as outcomes of sustainment. The SMS was designed to identify and support both the unique requirements for improving sustainment for individual funding initiatives as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches, thereby bringing precision to monitoring the structures and processes for sustaining each prevention approach and improving the likelihood of achieving sustainment of any grantee’s prevention efforts, regardless of source of funding [38].
The main objective of the current study was to explore and identify dimensions of sustainability by generating quantitative items from qualitative content domains of sustainment outcomes and sustainability determinants and subjecting them to confirmatory factor analysis in order to discern their factor structure. Our goal was to develop a scale that focused on the determinants and outcomes of sustainment of prevention programs and initiatives, and to examine its factor structure, reliability, and discriminant validity.