Study design
We carried out a descriptive study using multiple-methods involving quantitative and qualitative evaluations of the implementation fidelity using the Conceptual Framework for Implementation Fidelity (CFIF).8 A multiple-method study uses both quantitative and qualitative techniques, as distinctly separate parts of one research program.13 Evaluation of the degree of adherence to the core components of the EMA strategy was carried out through the analysis of a self-administered survey of HPs, observations, and secondary data from the national screening information system (SITAM, for its initials in Spanish). The analysis of moderating factors was carried out through semi-structured interviews with key stakeholders, and analysis of field notes that provided complementary information on different factors that affected the fidelity of implementation. Phases of EMA strategy implementation and evaluation are showed in Fig. 1.
Setting
The study was carried out in La Matanza, a district in the Metropolitan Area of Buenos Aires (Map 1). La Matanza has 2 million inhabitants, 50% of which are poor. La Matanza public health system is comprised of a network of hospitals and primary health care centers. For the uninsured population health services are provided free of cost, including screening, diagnosis, and treatment. In La Matanza, programmatic population-based self-collection was initiated in 2017. HPV testing (Hybrid Capture 2; Qiagen, Germantown, MD, USA) was introduced as primary screening for women aged 30 and older attending the public health system. HPV self-collection is offered by health promoters (HPs) during home visits. Unlike CHWs in Jujuy, HPs did not belong to the permanent staff of the health system; they were women from the community who receive social plans (conditional income transfer for protection of families in poverty conditions) provided by the Social Development National Ministry.
Conceptual Framework for Implementation Fidelity
The CFIF8 was used as a conceptual model to retrospectively evaluate the implementation fidelity of EMA strategy. We chose the CFIF because it is particularly appropriate to evaluate implementation fidelity of complex interventions -as the EMA strategy- because it allows a comprehensive assessment of different dimensions of implementation fidelity and the moderating factors that may influence it (Fig. 2). This model was integrated in all stages of the research, including conceptualization (e.g., selecting implementation components on which to focus), data collection (e.g., using components of the conceptual model to design interview guides), and analysis.
Several terms have been used interchangeably with fidelity in the literature, including treatment fidelity,14–17 intervention fidelity,18–19 implementation fidelity,8,9 or fidelity of implementation.20 This variety in terms associated with fidelity is related to the fact that initially, fidelity was mostly referred to the evaluation of what was delivered clinically in randomized trials. At present, it applies to all types of studies, from tightly controlled efficacy trials to implementation studies, although the focus on fidelity varies across each type of study.10 Following Carroll et al.,8 we have chosen the term “implementation fidelity” defined as the degree to which the EMA strategy was implemented as it was intended in the original model. It relates to the process of implementation of core strategy components.21 Implementation fidelity is critical to a successful translation of evidence-based strategies into real-contexts, and it is a factor that may influence the relationship between the strategy and its intended outcomes. Evaluation of implementation fidelity is particularly important given the greater potential for inconsistencies in implementation of a strategy in the real world rather than in experimental conditions.8
For Carroll et al.,8 the measurement of implementation fidelity is the measurement of adherence. Adherence includes the subcategories of content (i.e., 'active ingredients), frequency, duration, and coverage (i.e., dose). The content of the intervention may be seen as its 'active ingredients' or components that are essential to achieve the indented outcomes. In our case study, the content was defined as the core components of the EMA strategy (which are described below). Coverage refers to the degree to which women who met inclusion criteria accepted the intervention; and frequency and duration refer to whether the intervention/strategy was delivered with the regularity and duration as planned by its designers. In addition, the level of implementation fidelity achieved may be influenced by other variables called moderating factors. The original model described four moderating factors: intervention complexity, facilitation strategies, quality of delivery, and participant responsiveness. The CFIF model used in this study has been modified based on the adaptation made by Hasson et al.9,22 for the assessment of fidelity of complex interventions. Following Hasson´s model we have included the following moderating factors:
Participant responsiveness
refers to how well participants respond to or are engaged by the intervention. In our case study, responsiveness refers to the engagement of health staff involved in EMA strategy implementation (e.g., HPs that offer SC).
Intervention complexity
refers both to intervention and its implementation strategy characteristics – e.g., number of core components, – and to the way in which they have been described and transmitted to the implementers.8
Facilitation strategies
refers to support strategies that may be used both to optimize and to standardize implementation fidelity, i.e., to ensure that everyone is receiving the same training with the aim that the delivery of the intervention is as uniform as possible (e.g., manuals, monitoring and feedback, etc.).8
Context
refers to surrounding social systems such as structures and cultures of organizations and groups, inter-organizational linkages, and historical as well as concurrent events.9,14
The CFIF states that different moderating factors might affect, positively or negatively, the implementation process and its fidelity. In addition, factors interact with each other, and the effect of one factor on fidelity might be influenced by another moderating factor. 8
Description of the EMA strategy
The EMA strategy is a complex evidence-based intervention that was implemented, evaluated and scaled-up nested into the Jujuy Demonstration Project (JDP),23 -more details about the EMA strategy can be found elsewhere-.3,11 It is consisting of several active ingredients that address multifaceted processes within interpersonal, organizational, and community contexts.24,25 It is based on the concept of Cancer Care Continuum defined as a process of care consisting of several steps (screening-diagnosis-treatment) and interfaces between patients, providers and organizations. Central to the process of care across the continuum is the transfer of information and responsibility from one institution to another, from one health professional to another and from providers to patients.26 The EMA strategy involves a door-to-door offer to women of HPV self-collection by trained health staff together with provision of information about how to perform self-collection, sampling handling and transport of samples to the HPV laboratory, follow up and treatment of HPV-positive women at health centers (Fig. 3).11 These components are essential to achieve high level of adherence to screening, triage, diagnosis and treatment, the necessary steps to assure the screening program effectiveness to prevent CC. CFIF dimensions applied to the EMA strategy are as follows:
Content
As mentioned above, the EMA strategy included four core components that were defined as the content of the intervention. Table 1 shows specifications for each core component following Proctor et al. model.25
Table 1
Specification of core component of EMA strategy following Proctor model (Cont.)25
Core component: TRAINING
|
Actor
|
National and Local Training team expert in Cervical Cancer Prevention, HPV test and Health Communication. Experts that participated in EMA Study (RCT and Scaling Up in Jujuy province).
|
Actions
|
To provide training through different techniques (with expert presentations, discussions in small groups, and role playing) regarding: Cervical cancer prevention and HPV, HPV results.
To evaluate knowledge acquired during training (self-administered survey)
|
Target of the action
|
Health promoters in charge of Offer of Self Collection, Health supervisors, health professionals involved in EMA strategy implementation in La Matanza.
|
Temporality
|
Pre-implementation phase
|
Dose
|
Two workshops
|
Implementation outcome
|
Adoption and Fidelity
|
Justification
|
Health education/training of implementers
|
Core component: OFFER OF SELF-COLLECTION
|
Actor
|
Trained Health Promoters
|
Actions
|
To identify the target population: age, pregnancy, previous screening.
To provide information about HPV test and Cervical Cancer
To explain on how to perform self-collection
|
Target of the action
|
Women aged 30 and over
|
Temporality
|
Implementation phase
|
Dose
|
10 minutes step by step explanation of how to do self-collection
|
Implementation outcome
|
Acceptability and Fidelity
|
Justification
|
Door to door offer is effective to increase screening uptake and acceptability of HPV self-collection. 3–6
|
Core component: Sample Handling and Transport
|
Actor
|
Health promoters who offer self-collection /Supervisors/ health professionals involved cervical cancer prevention program
|
Actions
|
To label collectors with the woman’s name and the national unique identification number.
To fill in sample collection forms.
To transport samples at room temperature to primary health care centers.
To assure that specimens arrive to HPV laboratory within 14 days after sample collection.
|
Target of the action
|
Women aged 30 + who performed HPV Self-collection
|
Temporality
|
Implementation phase:
Labelling collector: during self-collection offer.
Transporting: within 14 days after sample collection.
|
Dose
|
Always
|
Implementation outcome
|
Fidelity
|
Justification
|
Protocol of identification and transport of samples following manufacture instructions.
|
Training
Self-collection should be offered by trained HPs. Training should include at least: 1) two workshops about CC prevention and HPV test, with expert presentations, discussions in small groups, and role playing to recreate different scenarios during the offer (Box 1), and 2) at the end of the last workshop, inviting HPs to complete a self-administered survey to evaluate knowledge acquired during training.
Box 1: Description of training workshops
Sections
|
Description
|
Project background
|
Provides information about the EMA study and the scaling-up into the programmatic context.
|
Cervical cancer
|
Provides scientific information about cervical cancer and its relation with HPV.
|
HPV-testing
|
Provides basic information regarding HPV testing as primary screening for cervical cancer prevention.
|
HPV Self-collection
|
Provides specific information about HPV self-collection: differences with clinician collected tests, steps of self-collection take-up, understanding results of self-collection and follow-up of HPV+ women.
|
Communication skills
|
Provides communication skills to conduct the educational talk (instruct women on how to perform self-collection).
|
Logistical procedures
|
Training about how to label and transport samples.
|
Adapted from Arrossi et al., 2017.11
Box 2. Description of core components, sub-dimension, items and activities included in the checklist
Core component: Offer of self-collection
|
Sub-dimension: Identification of target population (3 items)
|
Age
|
Pregnancy
|
Previous HPV screening
|
Sub-dimension: Key information during SC offer (7 items)
|
Information about HPV test
|
Information about CC prevention
|
Information about SC
|
Information about SC is painless
|
Information about possible results (HPV positive/negative)
|
Information about results delivery date
|
Information about results delivery methods
|
Sub-dimension: Step-by step explanation on how to perform self-collection (6 items)
|
Private place to perform SC
|
Information about different positions to perform self-collection
|
Explanation about that woman should be careful with the liquid inside the tube
|
Explanation about that woman had to insert the brush into their vagina until it reaches the bottom
|
Explanation about that woman had to rotate the brush 3 times.
|
Use of communication support material
|
Core component: Sample Handling and Transportation (4 activities)
|
To Check that the tube was correctly closed
|
To carry the tube in vertical position
|
To label collectors
|
To fill HPV-form
|
Offer of self-collection
Self-collection offered during home visits by trained HPs should include at least: 1) identifying the target population, 2) the offer of self-collection, 3) providing women with information about CC prevention and HPV self-collection (Box 2), 4) offering women to perform HPV self-collection, and a 10-min step-by-step explanation on how to perform it using communication support material (Additional file 2), and 5) providing women with the HPV collector to collect the sample.
Sample handling and transportation
Sample handling and transportation should include at least: 1) labelling collectors with the woman’s name and the national unique identification number, 2) filling in sample collection forms 3) samples transported at room temperature to primary health care centers and then to the HPV laboratory, 4) assuring that specimens arrive to the HPV laboratory within 14 days after sample collection, and 5) samples without liquid, brush, or identification data discarded at the HPV laboratory.
Follow-up and treatment
Follow-up and treatment should be organized according to national guidelines:27 1) HPV-positive women should be referred to cytology triage. HPV-positive women with normal cytology should repeat the HPV-test in 18 months 2) Women with abnormal cytology (ASCUS+) should be referred to colposcopy, and biopsy if needed, 3) Identified cases of CIN2 + should be treated according to standard protocols (loop electrosurgical excision procedure -LEEP- or conization and 4) HPV-negative women should be repeat screening in 5 years.
Dose (Duration)
The “dose” of EMA strategy was defined based on the expected duration of the step- by step explanation on how to perform self-collection (10 minutes).
Coverage (acceptability)
Coverage was defined as the proportion of women who accepted self-collection after the offer of self-collection. The expected acceptability of self-collection was based on results of EMA Study (86%).
Indicators and sources of data used to measure adherence (content, dose, and coverage) to the core components of the EMA strategy are presented in Table 2.
Table 2. Adherence to core component of EMA strategy: Indicators and source of data
ADHERENCE
|
SUB-DIMENSIONS
|
INDICATORS
|
SOURCE OF DATA
|
Content: core components of the strategy
|
TRAINING
|
Training
|
Number of planned workshops implemented
|
Training registries
|
Inclusion of expert presentations, role-playing and discussions in small groups
|
Training material
|
Participation in training
|
% of HPs that participated in the workshops
|
Training registries
|
Level of knowledge about the strategy among HPs
|
% of HPs with adequate knowledge about the EMA strategy
|
Self-administered questionaries after training
|
OFFER OF HPV SELF-COLLECTION
|
Place of offering SC
|
% of SC offered during home visit
|
Checklist
|
Identification of target population
|
Age
|
% of SC offers in which HPs asked the woman their age
|
Checklist
|
Pregnancy
|
% of SC offers which HPs asked if the woman was pregnant
|
Checklist
|
Previous screening
|
% of SC offers in which HPs asked if the woman had a previous HPV test
|
Checklist
|
Key information during SC offer
|
Information about HPV test
|
% of SC offers in which HPs mentioned information about HPV test
|
Checklist
|
Information about CC prevention
|
% of SC offers in which HPs mentioned information about CC prevention
|
Checklist
|
Information about HPV SC
|
% of SC offers in which HPs mentioned information about
|
Checklist
|
Information about SC is painless
|
% of SC offers in which HPs mentioned that SC is painless?
|
Checklist
|
Information about possible results (HPV positive/negative)
|
% of SC offers in which HPs mentioned information about possible HPV results
|
Checklist
|
Information about results delivery date
|
% of SC offers in which HPs mentioned information about results delivery date
|
Checklist
|
Information about results delivery methods
|
% of SC offers in which HPs mentioned information about results delivery methods
|
Checklist
|
Data collection
Quantitative data sources
Observations
To evaluate the implementation fidelity of the core components “Offer of self-collection” and “Sample Handling and Transportation” we conducted observations during routine self-collection offers. Between June 14 and July 23, 2019, four trained observers carried out 74 observations. All the HPs of La Matanza who received training in 2017 and offered self-collection in 2019 were eligible to participate. In total, 78 HPs who were working in the PHC system were eligible. Using a computer-generated random number list, we selected a sample of 20 HPs to be accompanied during a workday. An observation protocol (checklist) was developed based on the competencies and activities carried out by the HPs. We first identified a comprehensive list of planned activities by reviewing programmatic documents and training materials. We then classified this list of activities according to the content of the strategy (Box 2). The list of activities was validated by the training coordinator of the NPCCP. For each planned activity we evaluated whether it was implemented according to the EMA model (Yes/No). In addition, observers registered any adaptation of the offer.
Self-administered questionnaires
An ad-hoc self-administered survey was developed to evaluate knowledge acquired by HPs in training workshops. At the end of the last workshop, HPs were asked to complete an anonymous, self-administered survey. In total 171 HPs completed the survey. The self-administered survey included questions regarding scientific data on CC and its relationship with HPV, basic information on HPV testing, the step-by-step offer of self-collection, and follow-up of HPV + women. Results were registered in a specific database for processing and analysis.
National screening information system (SITAM)
We used data extracted from SITAM to evaluate the core component “Follow up and treatment” We analyzed the SITAM database containing records of all women aged 30 years and older screened in La Matanza using HPV self-collection during 2017–2018, and data recorded until December 2019 for follow-up. Colposcopies, biopsies and treatments not registered in SITAM were considered lost to follow-up. The data were accessed by authorized healthcare workers and researchers.
Qualitative data sources
During April-June 2020 we carried out six semi-structured interviews with key informants to explore their views on the EMA strategy and moderating factors that affect its implementation. Due to COVID-19 pandemics these interviews were carried out online. Online tools for data collection are suitable for different topics and allowed us to solve logistical issues.29 In addition, we analyzed field notes that were taken during observations, and training materials (e.g., power point presentations and training documents).
Data analysis
Quantitative data
Indicators presented in Table 2 were analyzed through frequencies and percentages. The percentage of implemented activities was calculated as follows: Total number of implemented activities /total number of planned activities following EMA model * 100. The percentage of implemented activities represented the degree of fidelity. Based on other studies that evaluate implementation fidelity in community settings 30 the following scoring categories were used in this study: 80–100%, high; 79–51%, moderate; and ≤ 50%, low.
We also calculated the percentage of adequate knowledge, defined as percentage of correct answers in the self-administered survey as follows: number of correct answers / total number of answers (adequate knowledge > 70% correct answers).
Qualitative data
Qualitative data were analyzed thematically31 using the CFIF dimensions. Two researchers became independently familiar with the data through audios and transcriptions and classified data using an initial codebook, to later compare and generate themes,32 debate, and resolve the inconsistencies with the other team members. We sought quotation examples that adequately graphed each theme and were the most relevant to assess implementation fidelity.
Stakeholder Engagement
Municipal health authorities and professionals actively participated in the planning, implementation, and fidelity evaluation of the strategy. The Director of Training of the Primary Health Care Direction of La Matanza was co-researcher. She participated in the design, implementation, and data analysis. All the staff of the Secretariat of Health helped organize the field work and participated in meetings where several aspects of the project were discussed. The Secretary of Health of La Matanza, health authority in charge of the primary health care centers, gave his support to the project. In addition, NPCCP staff participated in the design, checklist elaboration, field work, and data analysis. Also, Argentina NCI funded the Study.