The IKT approach in CEBHA+ can be conceptualized as an intervention within a complex social system. By implementing a systematic approach to engaging with stakeholders, building new or strengthening existing relationships, we aim to disrupt traditional knowledge translation approaches (26). We do this by redistributing and transforming resources, for example by having dedicated personnel for IKT ("IKT focal points”, one or two staff per site) that coordinate and implement local IKT efforts. The IKT focal points and the IKT evaluation team at LMU Munich make up the CEBHA+ IKT team.
At each CEBHA+ site, researchers engage with stakeholders. In the context of IKT in healthcare and public health, a concerned stakeholder can be defined as an entity responsible, involved or affected by health-related decisions that can be informed by research evidence (27, 28). Entities can be individuals, organisations, groups or networks operating at a local, regional, national or supra-national level. They can comprise the public, civil society organizations, the media, public health practitioners and service providers as well as public health policy-makers and industry actors across multiple sectors (29). These entities can be directly or indirectly affected – in positive or negative ways – by an effort or the actions of an entity (30). We target decision-makers, i.e. policy and practice stakeholders holding a position that allows them to make system-level, organizational or technical decisions that affect the general health of communities or populations. These decisions can be made on a more political or a more technical (i.e. programmatic) level within the health sector as well as in other sectors. Our primarily targeted stakeholder group might be extended as we proceed with the CEBHA+ IKT approach.
Initially, the IKT evaluation team at LMU Munich conducted a literature review on IKT, its mechanisms and effectiveness. We conducted forward citation tracking from a recent scoping review (19). This search (Feb 2018) did not yield updates of the review or more recently published evaluations of IKT approaches. Informed by the literature review and discussions with the IKT focal points, we developed a programme theory of the overarching CEBHA+ IKT approach. This programme theory was operationalized into a menu of options of IKT activities that researchers and stakeholders could adapt to their respective research tasks and contexts.
The overall CEBHA+ IKT approach comprises six steps, of which four are site-specific and include development of IKT strategies that are tailored to each CEBHA+ site. These site-specific IKT strategies recognise existing relationships at the different institutions, the local context, and the varying needs of different research tasks within CEBHA+. A coordinated IKT approach ensures that site-specific IKT strategies are harmonised to the extent reasonable. In terms of activities, the six steps towards a coordinated IKT approach in CEBHA+ are described in Table 1 and visualized in Figure 1.
Table 1
Overview of Development, Implementation and Evaluation of the Integrated Knowledge Translation Approach in CEBHA+
Step
|
Date
|
Description
|
Step 1
|
Nov 2018
|
In a foundational workshop developed and implemented by the Centre for Evidence Based Health Care, Stellenbosch University (31), one of the South African CEBHA+ partners, IKT focal points from each CEBHA+ site were introduced to the concepts of evidence-informed decision-making, IKT, and the overarching CEBHA+ IKT approach. Participants were introduced to the programme theory and received practical training on how to develop an IKT strategy (stakeholder mapping, stakeholder analysis, stakeholder engagement strategies) and draft a site-specific IKT strategy.
|
Step 2
|
Nov 2018
|
Workshop participants acted as multipliers within their country teams and shared their knowledge and skills gained during the workshop with the rest of their team members. By using the tools introduced during the workshop, country teams were requested to jointly agree on priority stakeholders with whom to engage over the course of the CEBHA+ project and to refine the site-specific IKT strategy drafted during the foundational workshop.
|
Step 3
|
Nov 2018 – Jan 2019
|
Each country team consulted priority stakeholders to introduce CEBHA+ and IKT and to gauge their interest in the research topic as well as their preferences for engaging with the CEBHA+ research team.
|
Step 4
|
Feb 2019
|
Country teams met to review and synthesise the information obtained in previous steps and to finalise the site-specific stakeholder analysis and IKT strategy.
|
Step 5
|
Feb 2019 – Dec 2022
|
Country teams are implementing and refining their site-specific IKT strategies and monitoring the indicators they defined as part of their site-specific IKT strategy.
|
Step 6
|
Feb 2020 – Jul 2022
|
A two-step evaluation, by LMU Munich, of the coordinated IKT approach and the site-specific IKT strategies will assess the added value of the IKT approach in CEBHA+ (stage one ongoing, stage two scheduled for Mid-2022)
|
Captions to Figure 1: This figure visualizes the process of developing, implementing and evaluating the Integrated Knowledge Translation Approach in CEBHA+. The dark shaded boxes describe steps which were pursued at the level of the research consortium, while the light shaded boxes describe steps pursued at the respective sites. Step 6 (evaluation) is pursued at both the consortium and individual site level.
These steps are not necessarily pursued in a linear manner across all sites but require a certain degree of iteration, i.e. steps 2 to 5 may be repeated as the stakeholder landscape changes. Ideally, stakeholders are engaged in the research process from an early stage (11). This engagement can comprise shaping the research questions, deciding on the methodology, involvement in data collection and development of tools, interpreting the findings and supporting dissemination of research results at the research task level (2). At the level of the CEBHA+ consortium, this “IKT thinking” has been embraced since the initial development of the project proposal, where relevant stakeholders from the respective partner countries were included in the process of setting research priorities. Over the course of the project, practical experiences with the development and implementation of site-specific IKT strategies, insights gained from monitoring and evaluation, or new scientific evidence may lead to adaptations in the overall IKT approach and/or the site-specific IKT strategies.
Programme theory
In order to guide the development and evaluation of the CEBHA+ IKT approach, we developed a comprehensive programme theory (see Figure 2). This builds upon two main frameworks: the Context and Implementation of Complex Interventions (CICI) framework (32) and a visual representation of IKT approaches, influencing factors, and outcomes as described in a scoping review by Gagliardi and colleagues (19). The CICI framework was used to structure the context and implementation component of the evaluation by providing a priori categories which informed the development of both the qualitative interview guide and the survey. The framework by Gagliardi provides an overview of empirical evaluations of IKT approaches, relevant enablers and barriers, preconditions, and outcomes. Moreover, it flags relevant and partly validated evaluation tools which informed the development of our survey. The contributions of these two frameworks were integrated to show how the CEBHA+ IKT approach is assumed to increase the use of contextualized research evidence in policy and practice decision-making.
As shown in Figure 2, the programme theory describes the coordinated IKT approach as an intervention implemented across the CEBHA+ consortium. In CEBHA+, a coordinated IKT approach is understood as an iterative process in which 1) relationship building and strengthening, 2) capacity building, and 3) collaborative research between CEBHA+ researchers and their priority stakeholders represent integral components of the intervention. Capacity building, relationship building, and their maintenance and collaborative research are expected to take place in an interactive manner and repeatedly over time, thereby reinforcing each other (3).
The implementation of these intervention components is expected to result in several intervention outcomes. The literature indicates that strong relationships increase mutual understanding (e.g. of language, work style, needs, constraints) and trust among researchers and decision-makers engaging in a partnership and that these influence attitudes towards the partners’ professional environment (i.e. attitudes towards research among stakeholders versus attitudes towards the policy-and-practice field among researchers) (19, 33-36). With this in mind, CEBHA+ researchers seek to build and enhance relationships with priority stakeholders. The experience of conducting collaborative research would ideally lead to appreciation for the collaborative process (33), promote a more diverse range of partners to engage in, and encourage involvement throughout the research process. Mutual capacity-building can occur in the form of improved access to information and/or contacts due to the research partnership, broadened perspectives and skills (e.g. research skills), and improved capacity for collaboration (33-35).
Moving on to intermediate outcomes, successful research partnerships between researchers and stakeholders is assumed to result in an increased perceived value of research evidence in terms of relevance, better applicability to the stakeholders’ field of activity as well as credibility. While both actual and perceived value are potentially affected by IKT efforts, studies show the individual perception of value has greater behavioural implications than the actual value of research evidence (37). We therefore focus on the perceived value of CEBHA+ research evidence with respect to its importance to the actual consideration and utilization of research evidence in decision-making. In turn, research evidence that is perceived as credible, relevant, and usable is likely to result in an increased intention to use that evidence in policy and practice decision-making. Subsequently, the intention to use research evidence is hypothesised to lead to the actual consideration of findings in a specific decision in another step towards the final outcome: the use of contextualized research in policy and practice decision-making.
Evaluation of the CEBHA+ IKT approach
Overview of evaluation approach
This evaluation is theory-driven. The programme theory provides the main structure for the evaluation and informs our choice of methods for data collection as well as data analysis. A comparative case study (CCS) design (38) will be used, including mixed methods to facilitate in-depth examination of IKT evaluation in each site. Each site (Ethiopia, Malawi, Rwanda, South Africa, Uganda) is considered a case. A CCS is undertaken over time and emphasizes comparison within and across contexts and thus allows to explore, understand, and explain how context influences the success of an intervention (39). Studying each case with its distinct IKT strategy at two time points will provide an opportunity to compare and contrast findings within and across sites.
As part of the process evaluation, we will look at how and why IKT works in the respective contexts. In line with the CICI framework, we will look at the different stages in the implementation process, the diverse implementation strategies employed in different sites and the respective implementation agents (32). We will moreover utilize our monitoring to deepen our understanding of the implementation process. This may be complemented by including quantitative implementation outcomes, such as acceptability and feasibility in the late-stage evaluation. As part of the outcome evaluation, we will investigate the intervention, intermediate, and final outcomes, as defined in the programme theory.
As shown in Figure 3, the evaluation process will comprise two phases, i.e. an early-stage assessment (early 2020) and a late-stage assessment (early 2022). In addition to the data analyses performed at the end of each phase, we will conduct an integrated analysis after the completion of phases 1 and 2 (until mid-2022). The evaluation process will comprise five different procedures: (a) quantitative surveys with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (b) qualitative interviews with all CEBHA+ priority stakeholders as well as CEBHA+ researchers; (c) continuous monitoring to capture ongoing IKT activities (mainly covered by the documentation of interactions between CEBHA+ researchers and policy-and-practice stakeholders reported in quarterly updates); (d) policy document analysis; and (e) integrated and cross-case analysis at the end of the evaluation. Each of the procedures will be conducted in all five CEBHA+ sites.
Data collection
Given the complexity of the initial programme theory and the multitude of outcomes along the causal path from the intervention to the intended outcome, we will use a combination of qualitative (semi-structured interviews, qualitative analysis of policy documents, IKT updates) and quantitative data collection approaches (survey). The survey is intended to touch upon all dimensions covered by the programme theory. The qualitative interviews, however, will allow us to explore how IKT exerts an influence in more depth. Also, we will focus on external context factors (macro-context) that impact on IKT, the process of IKT, as well as intermediate and final outcome(s). Two distinct surveys and two interview guides target researchers and stakeholders, respectively. This is necessary to reflect the different perspectives of individuals engaged in a partnership.
Development of data collection instruments
The underlying programme theory was used to develop survey and interview instruments. We initially constructed the survey based on suitable pre-existing instruments included in the review by Gagliardi et al. (2015). Literature searches were then conducted and instruments identified to inform the development of additional survey items (e.g. SAGE (40), PreVAiL (41), SEER (42)). Where items were non-existent, we constructed them de novo. The survey comprises both multiple choice and Likert scale questions. We provide an overview of included constructs and sub-constructs as well as their source in the additional files (Additional file 1).
The interview guides and surveys were pilot tested among two researchers (Rwanda and South Africa, one male, one female) who are not involved in the IKT implementation process. We assessed the tools for errors, comprehensibility, acceptability, and understandability. We were not able to pilot-test our tools with decision-makers from policy-and-practice due to their limited time availability and the high opportunity cost. Both the interview guide and survey were slightly adapted to account for the changed project context due to the COVID-19 pandemic and are available as additional files (Additional files 2 - 5).
Insights gained during the early-stage evaluation will inform the interview guides and surveys used in the late-stage evaluation. In addition to the topics covered in the early-stage assessment, participants will also be asked to describe how the research partnership has changed over time and to describe, where applicable, examples of successful translation of research evidence into policy or practice because of the research partnership.
Sampling
Selection of study participants for the surveys and interviews will reflect the following criteria:
- A CEBHA+ researcher or a priority stakeholder as identified by the respective CEBHA+ country team; AND
- actively involved in a research-stakeholder partnership in CEBHA+; AND
- 18 years or above; AND
- lives and works in the respective CEBHA+ country; AND
- is able to understand and articulate him/herself in English
With respect to sample size, we aim to interview five to ten participants per site, ideally both researchers and key stakeholders.
Recruitment
The evaluators will invite all eligible CEBHA+ researchers to participate in the study via email and will introduce the study. Eligible stakeholders will be invited to participate by their respective research counterparts and will be provided with the same information.
Participants willing to be enrolled will be contacted by the evaluators via email to arrange a date for the qualitative interview. A written informed consent form will be provided, detailing the study’s goals, estimated duration, scope, its voluntary nature, the pseudonymity and confidentiality of responses, and whom to contact regarding questions. Participants will be asked to return the digitally signed informed consent form or a photo/scan of the manually signed form. Participants will also receive an invitation link to access the online survey.
Survey procedures
The survey will be administered online using the LimeSurvey tool (https://www.limesurvey.org/en) and participants requested to complete the survey prior to the qualitative interviews. At the beginning of the questionnaire, participants will receive information regarding the study. Key information regarding the informed consent form will be reiterated. Participants who are not able to complete the survey before the qualitative interviews will receive reminders via email and orally during the interview.
Interview procedures
The interviews will be conducted by a member of the CEBHA+ evaluation team or trained research staff at the CEBHA+ site using an interview guide. Where feasible, notes will be taken by a second member of the CEBHA+ evaluation team or by trained research staff at the CEBHA+ site, who are not directly involved with the site-specific IKT strategy. We consider it important that the interviewer is not involved in the practical implementation of IKT at the respective CEBHA+ site and thus unknown to the interviewed stakeholder to reduce interviewer bias. All interviews will be conducted in English.
In light of the COVID-19 pandemic interviews will be conducted virtually (e.g. via skype, zoom or other) unless the interviewee insists on a face-to-face interview, the pandemic situation permits, and precautionary measures are in place. Where conducted in person, interviews will be carried out in a place of the interviewee’s choice, e.g. the office of the stakeholder or researcher. As a minimum requirement, the chosen place should be a separate room or another quiet place to avoid disturbances and outside observers. The estimated duration of a single interview is 30 to 60 minutes.
The interviewees will be asked to describe their perspective on the research partnership, contextual factors that hinder or facilitate the partnership or its outcomes, the expected benefits and ways to improve the partnership and/or outcomes. Interviews will be audio-recorded and transcribed verbatim. Interviewees will be given the opportunity to review the transcript, ask for modifications and give their approval for further use of the data.
Further data sources
Quarterly IKT updates
serve as a means of continuous exchange among the IKT teams. The updates take place during a virtual call and are facilitated by a structured reporting tool. This tool captures noteworthy interactions between researchers and stakeholders and their outputs. It also contains questions on the learnings and challenges encountered at each site. All updates are submitted to the evaluation team before the calls and will be analysed in Phase 1 and 2.
Policy documents
will be identified and analysed for the late-stage assessment. Relevant policy documents are those produced between 2017 and 2022 (i.e. the course of the CEBHA+ project) and on topics related to CEBHA+ research activities at the five research sites. Documents concerned with laws, regulations, plans and major programmes from different tiers of government will be selected. Other documents considered will include those issued by multi-sectoral working groups or semi-independent bodies with a mandate to devise strategies or plans. These policy documents will be identified through the quarterly IKT updates, interviews and stakeholder websites (incl. meeting agendas, minutes, protocols, or reports).
Data analysis
Descriptive analysis of survey data
All quantitative data will be analysed descriptively using IBM SPSS Statistics 27.0. Survey data will be analysed in conjunction with qualitative data for triangulation at the end of phase 1 and phase 2 (Figure 3).
Qualitative analysis of interviews
Interview transcripts will be analysed in MaxQDA using content analysis as described by Schreier (2012) (43). Codes will be developed both deductively (informed by the programme theory and resulting interview guide) and inductively (data driven).
For each of the sites, the interviews will be coded independently by two coders, with one of them being from the same site with a comparable cultural background as the interviewee in order to include both an insider as well as outsider perspective (44). Codes will be discussed among the two coders and adapted until agreement is reached. For each site, there will be two sets of category systems, one for the researcher and one for the stakeholder dataset. While we expect overlaps, there are likely to be distinctions. This will ultimately lead to a code book that will be applied to all interviews, while allowing for inductively developed, site-specific codes.
Document analysis of policy documents
The policy documents identified in the late-stage evaluation phase will be analysed through document analysis, a systematic procedure for reviewing or evaluating printed or electronic documents, which entails finding, selecting, appraising and synthesising data contained in the retrieved documents (45). We will look for specific indicators of CEBHA+-generated research evidence uptake into practice or policymaking. Qualitative content analysis will be applied to relevant sections of the document (45), by using an a priori developed list of categories which corresponds to the outcome indicators in our programme theory. Text passages that discuss any of these a priori categories will be coded according to this list, while others will be coded under new categories. The resulting category systems will be shared and discussed among the CEBHA+ IKT team members undertaking the document analyses.
Triangulation of data at each site
At the end of phases 1 and 2, qualitative and quantitative data will be integrated for each case using the triangulation method as described by Moran-Ellis and colleagues (2006), “following a thread” (46). Triangulation is a general approach whereby the convergence, complementarity, and dissonance of results on related research questions, obtained through different methodological approaches, sources, theoretical perspective, or researchers are explored. With this method, an initial analysis of multiple datasets (survey, interviews, quarterly IKT updates, policy document analysis) will be undertaken separately for each case (i.e. each site). At this stage, themes and analytical questions that require further investigation will be identified. These themes or questions – the thread – are followed throughout the datasets to capture corresponding, contradicting as well as lacking findings within cases. The final outcome of this step is to create concise case descriptions for each of the five individual CEBHA+ sites.
Cross-case analysis across sites
After data collection and triangulation (2022), we will conduct a cross-case analysis across findings from the five CEBHA+ sites, the two points of data collection and various methods of data collection. Cross-case analysis is a research method that facilitates the comparison of commonalities and differences in the events, activities, and processes that are the units of analyses in case studies. The aim is to compare key findings across cases (i.e. across sites). Drawing upon the programme theory, we will contrast what we found at each site with regards to the components of the programme theory. This cross-case analysis will also serve as basis for reflections on the community of IKT practice across sites within the CEBHA+ consortium. Insights from the cross-case analysis are likely to lead to refinements of the initial programme theory, leading to a middle-range theory.
Data Management
Data collection, retention and analysis will be conducted adhering to EU data protection regulations. All data (digital records, transcripts, interview notes, survey data) will be kept confidentially on a secured institutional IT infrastructure provided by the Institute for Medical Information Processing, Biometry, and Epidemiology (IBE) at the LMU Munich, which meets the requirements of an institute for research on highly sensitive patient and study data. For analysis, researchers from the respective CEBHA+ sites will have access to the pseudonymized interview data after transcription. All analyses will be undertaken in the cloud and no data will be stored locally at any of the sites.