Objectives
The primary objective of this multi-phase, mixed-methods project is to determine the feasibility and acceptability of implementing a RLHS in EIS. The secondary objective is to evaluate two-year impact of the RLHS on patient, family, EIS, and provincial-level outcomes (see Figure 1: Project conceptual framework).
More specifically, feasibility and acceptability are evaluated in terms of 2 objectives using the RE-AIM (Reach, Effectiveness, Adoption, Implementation and Maintenance) framework. Objective 1 investigates Reach, Adoption, Implementation, and Maintenance of: (i) a user-friendly electronic platform that captures continuous data on selected service quality indicators from individual EIS; (ii) continuous data-informed feedback to EIS; and (iii) data-informed and capacity-building activities tailored to EIS members of our LHS s and the overall Quebec EIS community for improving service quality where EIS components are weaker. Objective 2 addresses “Effectiveness” by evaluating improvements in (i) adherence to EIS components among participating EIS; (ii) capacity of EIS to collect data for monitoring quality of care; (iii) key patient and family outcomes; and (iv) program-level and provincial decision-making related to meeting quality of care standards in EIS.
Rapid Learning Health System (RLHS): A novel paradigm in EIS implementation
The new RLHS healthcare paradigm [27] has been shown to promote innovation and responsiveness by bridging the gap between evidence and practice, and improving efficiency, effectiveness, and quality in healthcare delivery, primarily in medical healthcare settings [28-32]. Among various definitions of RLHS [28-31], the most frequently cited is the Institute of Medicine definition which envisions “the development of a continuously learning health system in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience” [33]. According to the Institute of Medicine, a RLHS uses digital technologies to: (A) generate and apply the best evidence to support collaborative healthcare choices by patients and providers; (B) drive the discovery process as a natural outgrowth of patient care; and (C) ensure quality, safety, value, and innovation in health care [33, 34]. Digital technology, hardware and software that process and transmit digital information (e.g., electronic health records, databases, analytic tools, and visual dashboards), is at the core of the RLHS, providing data and information as catalysts for system “learning” and the transformation of clinical practice.
The RLHS addresses the knowledge-to-practice gap in medical care through the rapid and ethical transfer of knowledge produced by clinical research into routine clinical practice [35, 36]. The RLHS can foster a culture of shared responsibility between clinicians and patients [37, 38] and facilitate engagement by patients, clinical teams and program managers for the production and dissemination of evidence to the public [39]. A RLHS was thus chosen as an innovative research paradigm to guide transformation of the Quebec EIS system by addressing previously identified gaps such as lack of, or inconsistent monitoring of, quality and performance, and gaps between standards, evidence, and actual practice.
This study, conducted in partnership with EIS and key stakeholders, is grounded in principles of patient-oriented research that support meaningful and active engagement by patients and families. Adhering to this framework, we invited participation by patients, families, and knowledge users (e.g., program administrators, clinicians, and representatives of the Centre national d’excellence en santé mentale of the Quebec Ministry of Health and Social Services mental health advisory branch) in developing the study (e.g., study design, choice of outcomes) and we will continue this practice in the implementation and dissemination of study findings.
Guided by the literature [27, 33, 34], the implementation of our RLHS involves six iterative phases (AIM 1), as elaborated below (see Figure 2: RLHS for EIS for Psychosis). They are:
- Identify indicators for the RLHS for EIS through an external and internal scan and build the RLHS community. This involves knowledge synthesis of relevant peer-reviewed literature and EIS guidelines (external scan) and an environmental scan in the form of a survey for selected EIS (internal scan), followed by the selection of meaningful indicators for quality care in EIS.
- Design and set up a digital infrastructure for our RLHS to collect data routinely and iteratively regarding selected indicators of quality care in EIS.
- Implement the RLHS data capture platforms in real-life setting, while systematically and continuously analyzing data to generate new evidence and recommendations for improvement of the RLHS.
- Use RLHS digital technologies to collect data, perform analysis and propose recommendations for subsequent clinical care, as well as capacity-building activities tailored to evolving needs in individual EIS as identified by the data collected.
- Evaluate outcomes related to clinical practice and program-level changes.
- Evaluate overall outcomes of the RLHS and disseminate findings to key stakeholders.
Guided by the RE-AIM framework, we collect quantitative and qualitative data on Reach, Effectiveness Adoption, Implementation and Maintenance of the RLHS. These RE-AIM data will be analyzed to evaluate impact of the RLHS and address the two study objectives.
Study Settings
The RLHS literature recommends small scale pilot testing of digital technologies to build knowledge and confidence regarding complex digital systems, as such innovations are often viewed skeptically by healthcare clinicians and managers [27, 33, 34]. For this reason, we purposefully selected a maximum variation sample of 11 EIS among the 33 existing EIS in Quebec based on various characteristics: environment (academic and non-academic), setting (urban, semi-urban, and rural), years of operation (<5 versus >10 years) and patient age range covered by admission criteria (adolescence only, young adulthood only, or both) (see Table 1: Characteristics of the selected sites). EIS were also selected for their willingness to improve services and to represent diversity in relation to previously identified implementation challenges they have faced [21]. All 11 EIS invited to the study agreed to participate, although two of them mentioned staffing problems as a potential barrier to full participation in the project. These EIS were retained, as staffing is an important issue in real-world implementation. As “early adopters”, these EIS will guide implementation and future scale-up of the RLHS. Representatives of the 11 selected EIS participated in activities leading to development of this protocol and in project implementation activities.
Table 1: Characteristics of selected sites
EIS for Psychosis
|
Location of the EIS : Urban, semi-rural, rural
|
Years of operation
|
Average Number of active service users
|
Average number of full time staff
|
1
|
Urban
|
>10
|
290
|
16
|
2
|
Urban
|
>10
|
220
|
10
|
3
|
Urban
|
>10
|
150
|
12.5
|
4
|
Urban
|
>10
|
45
|
2
|
5
|
Urban
|
>10
|
270
|
14
|
6
|
Urban
|
>10
|
180
|
3
|
7
|
Urban – Semi-rural
|
<5
|
190
|
10
|
8
|
Semi-rural
|
>10
|
30
|
4
|
9
|
Semi-rural
|
<5
|
130
|
10
|
10
|
Semi-urban /Rural
|
>10
|
60
|
3
|
11
|
Urban /Semi-rural
|
<5
|
130
|
7.5
|
AIM 1: Assess feasibility and acceptability of a rapid learning health system (RLHS) in EIS
Phase 1: Identifying indicators for the RLHS for EIS through an external and internal scan and building the RLHS community (completed)
Quality indicators are measures or metrics based on guidelines or health organization directives used in monitoring quality of patient care [40, 41]. The research team identified indicators based on extensive literature reviews, including an external environmental scan of published national and international EIS guidelines and fidelity scales and the peer-reviewed literature on program evaluation and outcomes in EIS [20, 22, 42]. The team then conducted an internal environmental scan using an email survey (unpublished data) inviting clinicians, team leaders, local decision-makers/managers from participating EIS and other key stakeholders (service users, caregivers, researchers, and representatives from the Centre national d’excellence en santé mentale, Quebec Quebec Ministry of Health and Social Services), to prioritize the indicators by importance, document the degree of implementation for each indicator in their respective EIS, estimate capacity to improve implementation with the available resources and determine the availability and level of data already collected for each indicator. We also assessed what resources would be needed in each EIS for the measurement of designated indicators. Two group discussions by videoconference with representatives of the stakeholder groups representing the various EIS were convened to gather input and reach consensus on which indicators were most meaningful and feasible for data collection (see Figure 3: Involvement of stakeholders in our RLHS for EIS for Psychosis).
Table 2 provides the final list of evidence-based indicators and corresponding data collection procedures. In keeping with RLHS requirements, we chose measurable indicators (e.g., delay between referral and initial evaluation; a scale for self-rated clinical outcomes). The indicators were also chosen to balance maximum impact on program quality and patient outcomes with minimal burden related to data collection for the participating EIS.
Table 2: List of Indicators
Indicators
|
Example of data collected
|
Service users engagement and satisfaction with services
|
Services adapted to the young peoples’ need
Youth-friendly environment
Disengagement
Outreach practices
Youth satisfaction
|
Family engagement
|
Type of intervention offered
percentage of families reached
Number of visits
Satisfaction of family members/relatives
|
Access to care – pathways
|
Direct access
Referral sources, including self/community
Inclusion/exclusion criteria
Number of contacts before access
|
Access to care- systemic delays
|
Time between referral and
- The first contact
- The first assessment
- The start of treatment etc.
|
Continuous Education (CE)
|
Number and type of continuing education events attended by workers
Supervision/mentoring
|
Provider to patient ratios
|
Patient : Clinician ratios
|
Evidence based practices and recovery oriented
|
Cognitive Behavioral Therapy, family intervention, employment / study support, integrated treatment for Substance Use Disorders, peer support
Type of specialists who offer the interventions
Percentage of Injectable and Clozapine, etc.
|
Self-reported outcome by the patient
|
Patient’s evaluation of their health, recovery, quality of life
|
Phase 2: Designing the RLHS by building digital infrastructure (Completed)
Program-level indicator data are collected using the RedCap digital platform, which provides an open-access, user-friendly, secure electronic health data capture platform for routinely collecting real-time clinical data. Hosted by the Centre de Recherche du Centre Hospitalier de l’Université de Montréal, RedCap allows the team leader of each participating EIS to collect program-level data. The platform is accessible from any electronic device (e.g., computer, tablet, smartphone) through a secure, open access website. Each EIS can independently import data by answering specific multiple choice and open-ended questions on selected indicators.
Data from service users and family members on quality of services, an often-neglected indicator in the literature, are collected during onsite, virtual or outreach clinical appointments. Each participating EIS makes available a service user satisfaction questionnaire, on the “Happy Or Not” wireless digital terminals conveniently located (eg. on the walls of waiting rooms); alternatively, the questionnaire can be accessed on the web, or with any electronic device, through a barcode scan. The questionnaire includes three questions. The first asks, “Are you satisfied with the service you received today?” Using four smiley face emoticon buttons on the terminals, service users respond by choosing a face indicating whether they are “very happy”, “happy”, “unhappy”, or “very unhappy” with the service they received. The second asks, “Among the following items, which one did you appreciate the most/least: Quality of care and services, being welcomed with respect, feeling listened to, waiting time, respect for my opinion, something else”? These items were selected based on a literature review of youth-friendly mental health services and prioritized by consensus with service user representatives. Finally, comments are solicited using an open text box.
A second RedCap-supported digital questionnaire for more comprehensive evaluation of service quality and self-evaluation of personal recovery dimensions may be completed online or with any electronic device using a barcode scan or weblink. This quality of services digital questionnaire provides service users and family members with access to either the same questionnaire as the one on the service user feedback terminal with the four smiley emoticons (1-2 minutes duration), or a more detailed version (10 minutes duration). Satisfaction with the most recent onsite, virtual or outreach clinical appointment may be evaluated, as well as service users’ perceptions dating from the beginning of the EIS. Finally, service users can rate their satisfaction with their health situation and quality of life/recovery, and the impact of services on their recovery journey.
Phase 3: Implementing the RLHS data capture platforms (completed) and feedback development (ongoing)
New digital technologies (e.g., the RedCap digital platform for EIS clinicians, smiley feedback terminals, barcode scan and RedCap satisfaction with services questionnaire) were presented to key stakeholders for comments, and adjustments made, before deployment. The technologies were then tested with at least 2 representatives from each stakeholder group (service users, family members, EIS coordinators and managers) to ensure clarity of content and effectiveness of the digital tools. Learning from these usability testing activities is compiled and used by the RLHS project coordinator during virtual or onsite meetings with EIS managers, coordinators, or leaders to support easy, safe, and effective uptake of the RLHS. This implementation strategy leads to high program engagement and strengthens partnerships between researchers and experts, clinical staff, managers, and EIS leaders.
Phase 4: Using RLHS digital technologies to collect data, perform analysis and share results/feedback to EIS and all stakeholders (ongoing)
Data collection and support
The RLHS collects data on selected indicators from each participating EIS at four-month intervals, using the RedCap platform (see Table 2: List of indicators). Maintaining regular or as-needed contact with each EIS via web conferencing, telephone, email, or in person, the project coordinator supports participating EIS with data collection, use of the RedCap platform and integration of the collected data into clinical routines. EIS leaders and/or coordinators enter data on organizational indicators (e.g., number of clinical staff, caseload, referral sources) and evidence-based interventions offered (e.g., Cognitive Behavioral Therapy, supported employment, family interventions) directly into the RedCap digital platform.
Continuous real-time feedback to the EIS on quality indicators for essential components
After completion of the quarterly data collection cycle by the clinical team leader, the RLHS provides feedback to each EIS in the form of an individualized, user-friendly graphic report generated quarterly. Progress on specific program-level indicators can be tracked by each EIS over time and its implementation level compared with the aggregated data from other participating EIS, and with provincial standards. This feedback indicates whether the EIS meets, or does not meet, the provincial benchmarks for each specific indicator, providing the rationale for each essential component and guidance on how to improve implementation. The RLHS then uses the feedback system to guide subsequent actions toward better informed and evidence-based implementation [28]. Moreover, the aggregated data on service user and family perceptions of quality and satisfaction with services, including their self-assessments of progress toward clinical recovery, are integrated into the RedCap digital platform, allowing the RLHS to provide regular service user feedback to the individual EIS.
The EIS may receive feedback reports on services and the service user/family satisfaction ‘happy or not’ questionnaire by email or through a website, selecting a preferred frequency (e.g., daily, weekly, monthly). The EIS may also monitor their overall progress for selected time periods (e.g., daily, weekly, monthly, quarterly). These 2 reports may be used for administrative reporting, advocacy work to secure resources, guidance, and support for quality improvement in services, or for descriptions of clinical services tailored to service user, family, or other audiences.
Phase 5: Evaluating outcomes related to change at program-level based on capacity-building activities (Ongoing)
Capacity-building is understood as an evidence-driven process for strengthening abilities of individuals, organizations, and systems to perform core functions effectively, efficiently, and sustainably, continually improving and developing them over time [43]. The capacity-building activities are geared toward helping program managers, clinical team leaders and clinicians use data effectively to improve the quality of clinical practices, aligning them with guidelines and tailoring practices to data-identified program needs. These activities take the form of knowledge exchange events for improving knowledge and clinical skills, while providing program representatives and stakeholders with opportunities to share experiences, increase self-assessment skills and participate more fully in the RLHS.
The AQPPEP and the Centre national d’excellence en santé mentale of the Quebec Ministry of Health and Social Services have been partners in designing this project. Project-related webinars and online training with participating EIS occur roughly 3 times/year and are conducted with program leaders and/or coordinators and/or managers of each participating EIS. Clinical teams from participating EIS are met by the RLHS research team and/or representatives from the Centre national d’excellence en santé mentale to explain the project, examine their feedback reports in further detail, highlighting strengths and challenges of the EIS and to discuss the rationale behind essential components and alternative ways of reaching goals. In addition to these meetings involving all participating EIS, we partner with the Centre national d’excellence en santé mentale to provide individualized digital training and coaching to improve EIS performance on specific indicators and are developing an online media library for asynchronous training on related themes. Programs demonstrating high performance on certain indicators (positive deviance) may be partnered with programs needing help. The Centre national d’excellence en santé mentale, Ministry of Health and Social Services Ministry and the AQPPEP already use this type of system for peer mentorship. These approaches have proven effective for use in knowledge translation and implementation science [43-45]. A continuous back and forth between digital data capture, continuous feedback on performance, and capacity-building activities will facilitate positive evolution in aligning Quebec participating EIS with best practices.
Phase 6: Evaluating and disseminating RHLS outcomes to stakeholders (To be implemented)
The RLHS project and outcomes will be presented at AQPPEP events, which are attended by most staff from Quebec EIS, and at Quebec, Canadian and international scientific conferences. We plan to adapt the RLHS based on lessons learned from this pilot project in terms of successes, weaknesses, facilitators, and challenges. The anticipated longer-term structural impact of the project will be the adoption and integration of the RLHS by EIS across the province, ideally with support from the Ministry of Health and Social Services. The project will positively impact decision-making at the local and provincial levels to become more data-informed and responsive in real-time. Institutional bodies housing many of the EIS will be better able to monitor their implementation, targeting areas for improvement, resources needed, etc. The Ministry of Health and Social Services will be able to follow the progress of EIS implementation across the province in relation to changes in socio-political measures and context (e.g., investments, provision of new guidelines or revisions to existing guidelines). The AQPPEP, Ministry Centre national d’excellence en santé mentale of the Ministry of Health and Social Services, and similar organizations currently structured to train and support EIS will become more resource-efficient and effective after using the RLHS by tailoring their offerings to EIS, selecting appropriate target groups for training, and adopting data-driven evaluation and modification in capacity-building activities.
AIM 2: Assess impact of the RLHS in EIS
The RLHS will further provide us with valuable information and data suggesting whether this paradigm does indeed lead to improved quality of care in EIS. The RE-AIM framework, used to assess feasibility and impact in our project, was developed specifically to evaluate implementation of interventions in real-world settings, and sensitize researchers, knowledge users, and stakeholders to the essential elements involved in sustainable adoption and implementation of targeted interventions. We will assess Reach (what proportion of the targeted population participates in the RLHS), Effectiveness (impact of the RLHS on outcomes), Adoption (extent and ease of adoption of RLHS and degree of change), Implementation (performing in-depth analyses of the RLHS process data to determine facilitators and barriers that are associated with better implementation of RLHS), and Maintenance (extent to which the RLHS and its impact can be maintained) of this RLHS (see Figure 1).
To gain a qualitative perspective, we will invite all stakeholder groups (clinicians, managers, service users, family members), advisory committee members and representatives from the selected EIS (clinical staff, program leaders, managers, decision-makers) to participate in focus groups. Before the end of the project, a total of 5 remote focus groups (8-10 participants per group; 1.5 hours duration), will be implemented as follows: one group for clinicians, 2 for program leaders (one for medical, one for other professional team leaders), one for managers and decision-makers; one for service users and one for family members. Focus groups will be held by videoconference with a trained moderator and a research staff acting as co-facilitator. Focus group questions will be designed following Krueger and Casey [46] and structured to explore Lessard’s dimensions of learning health systems [37], which include five key dimensions that capture the nature of a RLHS: The goals pursued by a RLHS to promote evidence-based care and quality care; the social dimension focused on building a community; the technical dimension addressing digital data integration into routine care; the scientific dimension enabling learning, innovation, and discovery; and the ethics dimension ensuring that a RLHS pursues its learning and innovation activities in a manner that protects patient rights and privacy. Focus group participants will provide information on their experiences and perceptions related to the RLHS, impact of the RLHS for them, willingness to change and maintain the use of the RLHS, attitudes about data collection, and facilitators/barriers encountered in their implementation journeys, its impact on decision making both at clinical and administrative levels. A research assistant will transcribe the focus group audio files and prepare them for analysis. Informed by the Braun and Clark analytic procedure [47] we will: (a) familiarize ourselves with the data (reviewing transcriptions for accuracy); (b) generate initial codes using Lessard’s dimensions of learning health systems [37]; (c) review and redefine themes; and (d) further unpack the analysis through the writing process.
For a quantitative and qualitative picture of EIS evolution along the RE-AIM parameters, we will track uptake of the RLHS and extract data on all indicators from the RedCap platform, monitoring performance for each EIS on each indicator (see Table 2) and comparing data from baseline to project completion to assess effectiveness.
Reach will be assessed in terms of:
- Proportion of invited EIS that participate in the project
- Proportion of invited EIS representatives (e.g., clinicians, team leaders and/or managers) and invited service users and family members who participate in capacity-building activities, knowledge exchange events and implementation meetings.
- Proportion of participating EIS who adopt our electronic data capturing platform to ask service users and family members to provide information on satisfaction with services, self-evaluation of recovery dimensions and on the impact of services on recovery.
- The proportion of invited people from each stakeholder groups (clinicians, managers, service users, family members) who participate in research focus groups.
Effectiveness will be assessed in terms of:
- Improvement over time in indicators, e.g., reduction of delays in access, increase in service user and family member engagement in services, satisfaction with services and recovery outcomes (eg. employment).
- The increase over time in provision of evidence-based care as required by Ministry guidelines - the “cadre de référence” (e.g., proportion of EIS offering Cognitive Behavioral Therapy, family interventions, supported employment/education, integrated substance use disorder interventions, peer support and pharmacological interventions).
- The accuracy of data obtained from each EIS through the LHS electronic platform throughout the project, based on a comparison of the program-reported data on RedCap surveys in our RLHS with data collected by chart review on a selection of charts from each participating EIS. De-identified data on “access to care” (e.g., referral sources, delay from referral to initial evaluation); interventions offered (e.g., Cognitive Behavioral Therapy, family psychoeducation); and indicators of user engagement will be collected by research participants from the charts of 20 randomly selected service users at baseline and an additional 10 service users at all the other different timepoints (4 months preceding study onset and every 4 months subsequently until study completion). This step will ensure the trustworthiness of self-report data from the EIS by comparing self-report data with objective data from the files (e.g., delays for evaluation, percent of service users offered family interventions). If trustworthy, data reported by programs themselves, as in our RLHS, may enable the creation of large, ecologically valid datasets that may be used to draw inferences about program performance and its relationship to patient outcomes on different recovery dimensions.
- The perception of each stakeholder groups (clinicians, managers, service users, family members), of the ability of the RLHS to promote evidence-based care and quality of care in EIS.
Adoption will be estimated in terms of:
- The proportion of programs represented, and proportion of each invited stakeholders group (clinicians, team leaders, managers, service users and family members) in attendance at the different training sessions offered by the project.
- Number of programs not involved in the research project that express interest in adopting the RLHS after attending presentations at AQPPEP or other events.
- Progression over time in the proportion of data collected by program staff and service users, and completion rates.
- Proportion of participating EIS who continuously engage service users and family members to provide information on satisfaction with services, self-evaluation of recovery dimensions and on the impact of services on recovery using our electronic data capturing platform
- The perception of each stakeholder groups (researchers, clinicians, managers, service users, family members), of the ability of the RLHS to foster a learning community.
- The perception of each stakeholder groups (clinicians, managers, service users, family members), of the feasibility for EIS to integrate indicators/digital data into routine care
Implementation will be estimated in terms of:
- The extent to which capacity-building strategies (e.g., training) are implemented (target of at least one after each data 4-monthly data collection period),
- Proportion of participating programs using regularly the LHS health technologies over time until the end of the project.
- Barriers, facilitators, and overall burden related to implementation of RLHS, as assessed qualitatively in focus groups.
- The perception of each stakeholder groups (clinicians, managers, service users, family members), of the feasibility of implementing the RLHS in EIS.
- The perception of each stakeholder groups (clinicians, managers, service users, family members), of the extent to which the RLHS protects patient rights and privacy.
Maintenance, defined as use of health technologies over time, with regular data collection by programs estimated in terms of the extent to which data collection is sustained by the participating programs over the course of the project:
- Program commitment (e-survey) to continue using the electronic data capture system beyond the project.
- Proportion of EIS attending advisory committee meetings over the course of the project and until the end
- Proportion of EIS attending capacity-building/knowledge exchange events over the course of the project and until the end.
The perception of each stakeholder groups (clinicians, managers, service users, family members), on how the RLHS enables learning, innovation, and discovery.