Beyond Interviews and Focus Groups: Mapping New Ways to Integrate Innovative Qualitative Methods into Randomised Controlled Trials of Complex Public Health Interventions

doi:10.21203/rs.2.133/v1

Download PDF

Method Article

Beyond Interviews and Focus Groups: Mapping New Ways to Integrate Innovative Qualitative Methods into Randomised Controlled Trials of Complex Public Health Interventions

https://doi.org/10.21203/rs.2.133/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 06 Jun, 2019

Read the published version in Trials →

You are reading this older preprint version

Read the latest preprint version →

Background: Randomised controlled trials (RCTs) are widely used for establishing evidence of the effectiveness of interventions, yet public health interventions are often complex, posing specific challenges for RCTs. While there is increasing recognition that qualitative methods can and should be integrated into RCTs, few frameworks and practical guidance highlight which qualitative methods should be integrated and for what purposes. As a result, qualitative methods are often poorly or haphazardly integrated into existing trials and researchers rely heavily on standard interviews and focus group discussions. To improve current practice, we propose a framework for innovative qualitative research methods that can help address the challenges of RCTs for complex public health interventions.

Methods: We used a stepped approach to develop a practical framework for researchers, which included: (1) a systematic review of the innovative qualitative methods mentioned in the health literature, (2) in-depth interviews with 23 academics from different methodological backgrounds working on RCTs of public health interventions in 11 different countries, and (3) framework development and group consensus-building process.

Results: The findings are presented according to the CONSORT Statement categories for ease of use. The main challenges of RCTs for public health interventions are identified alongside each of the CONSORT categories and potential innovative qualitative methods that overcome each challenge are listed as part of a Framework for the Integration of Innovative Qualitative Methods into RCTs of Complex Health Interventions. Innovative qualitative methods raised in the interviews and discussed in detail include: rapid ethnographic appraisals, document analysis, diary methods, interactive voice responses and SMS, community mapping, spiral walks, public randomisation, pair interviews, visual participatory analysis, among others.

Conclusions: The findings of this study point to the usefulness of observational and participatory methods for trials of complex public health interventions, offering a novel contribution to the broader literature about the need for mixed methods approaches. Integrating a diverse toolkit of qualitative methods can enable appropriate adjustments during RCTs, which in turn create more sustainable and effective interventions. However, this will require a cultural shift including the adoption of method-neutral research approaches, transdisciplinary collaborations, and a shift in publishing regimes.

General Medicine

qualitative methods

complex interventions

public health

RCTs

innovation

In this article, we argue that many of the challenges facing randomised controlled trials (RCTs) of complex public health interventions can be addressed through the integration of a diverse toolbox of qualitative methods, and propose a framework of potential methods. RCTs are widely used for establishing evidence of the effectiveness of interventions, yet public health interventions are often complex, posing specific challenges for RCTs that qualitative research can help address. Qualitative research methods offer important insights for the evaluation of health interventions and are increasingly being used as part of RCTs [1-3]. Research investigating the potential value of qualitative methods has highlighted their role in facilitating the transferability of interventions, improving external validity, providing a more nuanced understanding of contexts and processes, and improving delivery of interventions. All of these ultimately increase the utility of evidence generated during RCTs [2-5].

Pope and Mays [6] claim that qualitative research is a “prerequisite of good quantitative research, particularly in areas that have received little previous investigation” (p42). It can provide insight into the contextual circumstances of the implementation, delivery and evaluation of interventions [7]. Qualitative research is particularly valuable for evaluating complex health interventions, which are often found in public health. A complex intervention is composed of different interacting components that are multifaceted and socially mediated [8, 9]. The complexity resides in the variety of behaviours required by participants in the intervention; the groups or organisational levels targeted; the potentially large number of outcomes; and the degree of flexibility permitted by the intervention [10]; all of which are associated with emergent phenomena that are difficult to predict. All these factors mean that complex health interventions are often challenging to define and therefore reproduce [11, 12]. Qualitative methods can help address these challenges by “reaching the parts that other methods cannot reach,” including aspects of behaviours, attitudes and interactions that affect the external validity and reproducibility of health interventions [6].

While there is increasing recognition that qualitative methods can and should be integrated into RCTs, few frameworks and practical guidance highlight which qualitative methods should be integrated and for what purposes. As a result, qualitative methods are often poorly or haphazardly integrated into existing trials [1, 5]. This is evidenced by the lack of explicit reference to how qualitative findings have been used to interpret or help explain quantitative trial results in published articles [3, 13]. It contributes in turn to variation in the quality of qualitative research used alongside trials [5], and is compounded by what O’Cathain and colleagues [3] refer to as the “add-on status of qualitative research” (p121) in quantitative health research.

Challenges of using RCTs to evaluate complex health interventions

Although RCTs are currently perceived as the ‘gold standard’ in global health evaluation, their limitations are well-documented [14, 15]. Power and colleagues [11] point to several implementation trials that have lost internal validity when failing to consider the broader contexts in which they were carried out [16-18]. Other scholars acknowledge a disconnect between the recommendation that interventions be standardised to ensure valid measurement of trial outcomes and the very concept of complex health care systems [14].

These critiques of RCTs and standardised approaches to intervention are well recognised [10]. Recent Medical Research Council (MRC) guidelines on developing and evaluating complex interventions suggest a range of designs to address the challenges of conventional experimental trials, including cluster randomised and stepped wedge designs; preference trials including ‘Wennberg’ and ‘Rucker’ designs, which base randomisation on individuals’ preferences; and randomised consent ‘Zelen’ designs, which randomise individuals prior to taking consent [19]. Others have discussed the need for ‘adaptive’ trial designs whereby changes are made to an intervention or trial based on information collected during the experiment phase [20]. An example of such adaptations is the use of ‘pragmatic trials,’ which attempt to establish the effectiveness of interventions under routine conditions instead of rigid experimental design [21]. Hawe and colleagues [22] argue that, in pragmatic trials the evaluation process and function can be standardised, while the form of the intervention is made adaptable to context. Pragmatic trials are designed to reflect the real-world heterogeneity of participants and situations, often keeping exclusion criteria to a minimum and not necessarily requiring blinding [23]. A pragmatic RCT approach is seen as providing an opportunity to evaluate not only the intervention, but also its interaction with the complex social, cultural, legal and ethical context, which may be as important as the intervention itself [23]. This is unlike conventional experimental designs in which the intervention is normally defined at the outset with little room for adaptation [21].

However, as large-scale experimental studies, both experimental and pragmatic RCTs remain resource intensive, posing challenges to produce reliable and accessible evidence of generalisable impact and relevance to the targeted population [24-26]. The persistent use of trials to evaluate complex interventions has been critiqued as ‘consistently becoming one of expensive failure – that is weak or non-significant findings at huge cost’ [22] (p1561). Chalmers and Glasziou argue that there is 85% cumulative waste in the production and reporting of research evidence, generating unnecessary losses in investments as a result of entirely correctable problems [24]. An important aspect of this waste relates to inappropriate methods, which contribute to incomplete reporting of trial outcomes and the need to measure new outcomes not part of the original plan [26].

In an attempt to address these limitations and drawing on the advantages of qualitative methods discussed previously, we propose a framework for integrating qualitative methods into quantitative evaluations of complex health interventions. We aim to answer the question: how can innovative qualitative methods address the challenges of RCTs as an evaluation methodology? In this article, we draw on key informant interviews with 23 researchers involved in RCTs to identify how qualitative methods can be used to address the challenges that the evaluation of complex health interventions raises for RCT methodologies.

Using qualitative methods to address the challenges

Methodological frameworks for the use of qualitative methods alongside quantitative intervention evaluation methods are currently limited in justifying the need for qualitative methods. Mixed methods evidence about the use of qualitative methods alongside trials centres on time-related or ‘temporal’ frameworks. Along these lines, Sandelowski [27] highlights how qualitative research can be incorporated either as formative evaluations before the trial begins, process evaluations during the trial, or impact evaluations after an intervention [27]. Creswell and colleagues build on Sandelowski’s work and argue that it is the purpose of the qualitative data collection that defines whether the “before,” “during” or “after” model should be used. In the stages before a trial begins, qualitative research is said to be most useful for defining research topics while ensuring the intervention’s relevance and appropriateness to the populations of interest [28]. Nested within a trial, qualitative process evaluations may indicate the reasons behind an intervention failure, unexpected consequences, or success [8]. Finally, after a trial is complete, the use of qualitative methods is said to help “interpret quantitative findings and question underlying theory and assumptions to better inform future hypotheses and intervention design” [29] (p714). Frameworks within this group include the MRC Framework [8], which also divides the process of developing and evaluating an intervention in a trial into time-related phases.

There are different approaches. Flemming et al. [28] have developed a framework that focuses on the contribution of qualitative research to specific RCT processes, including planning, recruitment, randomisation, conduct, attrition and implementation. In line with Flemming’s work, O’Cathain and colleagues [3] argue that temporal frameworks contribute to a lack of clarity about when the qualitative research actually takes place as part of a RCT. Synthesising the use of qualitative research within specific trials across 296 studies, they produce an alternative framework that includes five broad categories: the intervention being tested, the trial design and conduct, outcomes, measures used within the trial, and the intervention studied.

However, neither temporal nor process frameworks go much beyond a description of qualitative research and its characteristics. Despite methodological discussions about when to collect qualitative data and for what purpose, there has been little discussion to date of what qualitative methods might be most useful for complementing quantitative RCT data in the evaluation of complex health interventions. As such, existing frameworks have not contributed to broadening the discussion around the inherent value of having a diverse selection of qualitative methods to draw on for RCTs, and how much greater innovation and diversity of methods could be drawn on to strengthen what are often unrealised attempts at mixed methods. Thus there is a need for trialists to ‘look beyond focus groups and interviews’ and expand the extremely limited diversity of qualitative methods currently being used [3].

In this article, we offer a new framework developed from expert researchers’ accounts of using qualitative methods as a component of trial. Rather than offering an alternative to temporal or process frameworks, our aim is to highlight the diversity of qualitative methods that can be used to address many of the major challenges complex health interventions pose for RCTs. We suggest innovative examples of qualitative methods to demonstrate the adaptive and extensive nature of qualitative research as part and parcel of RCTs.

We followed a stepped approach to develop a practical framework for researchers selecting qualitative methods to use within RCTs of public health interventions. Our aim was to cover a range of potential trial designs, including individually randomised, cluster randomised and stepped wedge. Our approach included (1) a review of the qualitative methods mentioned in the health literature, (2) in-depth interviews with academics from different methodological backgrounds (quantitative and qualitative) working on RCTs of public health interventions, and (3) framework development and group consensus-building.

The inclusion of qualitative and quantitative researchers in the study provided an opportunity to gather methodological perspectives on the inherent value of using qualitative methods within trials. The use of in-depth interviews to gather academics’ opinions and experiences provided a means of overcoming some of the power dynamics involved in qualitative and quantitative research paradigms and minimising the influence of group dynamics [30]. The participants in the study were also invited to be co-authors on the final paper to ensure the framework was developed and refined through a group decision-making process.

Step 1: Review of Qualitative Methods used in RCTs

An initial list of innovative qualitative methods was derived through searching the published literature using the databases Pubmed and Scopus. We used the term ‘innovative’ to refer to qualitative methods other than standard interviews and focus group discussions. The “sources” section of Scopus was searched on 08/11/2017 for journals that contained “qualitative” (n=23) and “method” (n=160) in their titles. Journals were assessed for relevance based on their focus on qualitative methodologies and English language. This search produced a list of 25 journals (Table 1). Two further journals were added that had been identified by Wiles et al. [28].

Table 1. Final list of journals searched for innovative qualitative methods.

Behaviour Research Methods

BMS Bulletin of Sociological Methodology/Bulletin de Methodologie Sociologique

Cultural Studies - Critical Methodologies

Field Methods

Forum Qualitative Sozialforschung

International Journal of Qualitative Studies in Education

International Journal of Qualitative Studies on Health and Well-being

International Journal of Research and Method in Education

International Journal of Social Research Methodology: Theory and Practice

Journal of Mixed Methods Research

Methodology

Organizational Research Methods

Psychological Methods

Qualitative Health Research

Qualitative Inquiry

Qualitative Report

Qualitative Research

Qualitative Research in Psychology

Qualitative Research Journal

Qualitative Social Work

Qualitative Sociology Review

Qualitative Research in Sport, Exercise and Health

Sociological Methodology

Sociological Methods and Research

The International Journal of Qualitative Methods

Quality and Quantity

Methodological Innovations Online

The website catalogues of these journals were then searched with the terms “innov*, new, novel, emerg*” as previously used in the methodology by Wiles and colleagues [31]. These search terms were used in order to identify methods considered as innovative or new to qualitative methodologists. This produced 654 search results which were exported to Endnote (Thomson Reuters). Journal articles from before 2008 were removed to capture methodological discussions over the past 10 years. Abstracts were screened for mention of qualitative methodologies beyond interviews and focus group discussions, or for discussions of methodological innovations. This produced 127 full text articles of interest, which were further screened for mention or discussion of methods. A list of methods was subsequently compiled (Table 2).

Table 2. List of qualitative methods identified in the literature

Art	Art workshops, collage, dance, decoupage, drama, drawing, drawing method of storytelling, graffiti, imprography, imitation games, improvisation, magazine collage, mural, music, painting, performative methodologies, role-play, scenario workshop, sculpture, sketching, street-theatre.
Mapping	Argument maps, body-mapping, circle map, concept mapping, digital mind maps, digital traces, emotion map, process maps, egocentric sociograms, social network analysis, spider diagrams
Multimedia	Avatar representation, bio-photographic elicitation interviews, video recordings, computer mediated communication, conversation audio recordings, documentary film, head-mounted cameras, spatial montage, skype interviews, twitter data, video shadowing, videoconference focus groups, videovoice.
Narrative	Audio diaries, scrapbook diaries, biographic narrative, biographic workshop, creative non-fiction, creative writing, digital storytelling, dramatic writing, experimental writing, fiction writing, memory box, narrative poetry, poetic reflection, scroll-back method, stimulated recall.
Visual	Visual dialogues, flash card activity, interpretation panels, mood boards, photo elicitation, photographic portraits as autobiography, photography exhibition, photovoice, social vignettes.

Step 2: Key informant identification and in-depth interviews

Identifying key informants: The list of methods generated during Step 1 was adapted to produce a search strategy to identify scholars working in mixed-methods. We searched Scopus and Medline for articles that mentioned any of the qualitative methods from Step 1 alongside RCTs. Emphasis was placed on methods that were not basic interviews or focus group discussions without a theoretical rationale mentioned in their design. This produced a final selection of 9 journal articles (Figure 1). Authors of these articles were reviewed to categorise them as potential key informants according to whether they were primarily publishing qualitative or quantitative research, their discipline, geographic location, and research position [32].

The initial list of scholars identified in the literature (n=85) was reduced to first and last authors only (n=18), and supplemented by professional research networks (n=25) and snowballing (n=15). The final list of experts (n=65) and their categorisation was used to ensure maximum heterogeneity in the group composition as a means of encouraging and heightening creativity in the decision-making process that followed [33].

We invited 58 experts to participate in the study. In the invitation sent by email, they were given an information sheet and consent form, and asked to confirm their consent to participate with an explicit email response. Interview participants were offered the opportunity to contribute as authors of this paper, under the condition that they provide input into each draft. A final tally of 23 experts agreed to participate in the in-depth interviews (Table 3).

Table 3. Demographic characteristics of respondents

	n	%
Total	23	100
Primary methodological approach
Quantitative	4	17%
Qualitative	10	43%
Both	9	39%
Research position
Researcher (inc. associate and senior)	6	26%
Lecturer (inc. senior)	11	48%
Professor	6	26%
Primary area of expertise
Gender and HIV	2	9%
Maternal, child and neonatal health	4	17%
Health services	4	17%
Behavioural health	1	4%
Anthropology	5	22%
Epidemiology	4	17%
Sexual and Reproductive Health	2	9%
Methodology	1	4%
Region of affiliation
Europe	17	74%
North America	4	17%
Africa	2	9%
Primary region of research
Europe	3	13%
North America	1	4%
Asia	6	26%
Africa	8	35%
Not specific	4	17%
Method of identification
Literature review	3	13%
Network search	13	57%
Snowballing	7	30%

Participants had experience working in China, Ethiopia, India, Malawi, Nepal, Rwanda, South Africa, Uganda, UK, US, and Zambia. Anonymity was maintained between participants throughout the study up until the paper writing stage, and participant co-authors were not involved in the analysis of the raw data. The study received ethical approval from the UCL Research Ethics Committee number 12449/001.

Data collection through in-depth Interviews: A topic guide was used for the interview with a broad set of questions about the inherent value of using qualitative methods within RCTs, methods being used to address intervention or evaluation challenges, and some of the challenges faced when mixing methods. Interviews were conducted by phone or Skype and lasted 50 minutes on average. The interviews were audio recorded and transcribed. Transcripts were entered into NVivo (QSR International) for qualitative data analysis (QDA).

Step 3: Framework development and group consensus process

We conducted a thematic analysis of the interview data using a framework approach [34]. We adapted a version of the Consolidated Standards of Reporting Trials (CONSORT) Statement [35] as our framework in an attempt to move beyond current discussions of how qualitative research can add value to trials and highlight the indispensable nature of qualitative methods as a complement to quantitative research methodologies. CONSORT is recognised as an essential tool for reporting trial outcomes, and widely used by medical journals and international editorial groups [36]. This provided an ideal starting point for appealing to quantitative researchers using a well-recognised framework. To adapt the CONSORT Statement for our purposes, our research team (KD, NM, JM) conducted a rapid review of the literature on the main methodological challenges of using RCTs to evaluate complex public health interventions and organised these challenges according to the CONSORT Statement’s main headings (Table 4). We then merged them into categories in terms of their key challenges.

Table 4. Challenges identified in the literature by CONSORT Statement´s main headings [37-42]

CONSORT headings	Challenge	Threat	Merged categories
Objectives	Ensure relevance of the problem to the context	Can reduce engagement	Background and setting
Generalisability and applicability	Obtain buy-in from stakeholders	Can reduce implementation, sustainability and translation of results into practices	Background and setting
Participants	Guarantee representativeness of the sample	Selection bias: can reduce internal and external validity	Recruitment and enrolment
Sample size	Develop efficient recruitment methods	Selection bias: can reduce internal and external validity	Recruitment and enrolment
Intervention	Ensure cultural acceptability and practical feasibility	Adherence and withdrawal bias: can reduce internal validity	Intervention design and compliance
Implementation	Improve fidelity to the intervention	Adherence bias: can reduce internal validity	Intervention design and compliance
Randomisation	Guarantee comparability between groups	Selection and confounding bias: can reduce internal and external validity	Randomisation and allocation
Allocation	Reduce manipulation during the allocation	Selection and confounding bias: can reduce internal and external validity	Randomisation and allocation
Participant flow	Minimise the number of participants leaving the study	Attrition and confounding bias: can reduce internal validity	Participant follow-up
Data collection	Enhance reliability and quality of data	Information (instrument, recall and social desirability) bias: can reduce internal validity	Data collection
Data collection	Maintain objectivity during the data collection process		Data collection
Interpretation	Identify the mechanisms underpinning the effect of the intervention	N/A	Analysis and results

This provided the adapted framework for our thematic analysis of the in-depth interviews with scholars. Using this framework, we completed a thematic analysis in NVivo of the interview transcripts, aligning what scholars had said about the inherent usefulness of particular qualitative methods for the RCTs they had worked on with the RCT challenges in our framework. We then shared the draft framework with participants in two separate rounds in order to elicit feedback and facilitate group consensus on the alignment between RCT challenges and specific qualitative methods.

The aim of our analysis was to generate a list of innovative qualitative methods that help to address common methodological challenges of RCTs (Table 4). We present our results following the adapted CONSORT statement framework used to guide the analysis. Common methodological challenges are organised according to the merged categories, and potential qualitative methods that overcome each challenge are listed. Our results are presented according to seven categories: 1) Background and Setting; 2) Intervention design and compliance; 3) Participant recruitment and enrollment; 3) Allocation of the intervention and randomisation; 5) Participant follow-up; 6) Data collection; and 7) Analysis and results. Under each category, we have combined findings from the in-depth interviews and evidence from the literature review as a means of expanding the scope of the results and describing a more comprehensive methodological landscape.

Table 5: Framework for the integration of innovative qualitative methods into RCTs of complex health interventions

Categories adapted from CONSORT	Methodological Challenge	Threats to design or results	Type of bias	Validity affected	Qualitative research solution	Examples of qualitative methods
Background and setting	Ensure relevance of the problem to the context	Affects applicability, acceptability , sustainability and transferability of potential positive findings	N/A	External	To identify social, cultural, health, economic or political factors that might affect uptake and sustainability of positive results	Direct observation, ideally in-depth but if time or resources are limited, as part of a rapid ethnographic appraisal or Broad Brush Survey
Background and setting	Obtain buy-in from stakeholders	Reduces translation and sustainability of results into changes in policy and practices	N/A	External	To overcome potential barriers to implementation and promote uptake of intervention	In-depth interviews with policy makers and stakeholders \| Document analysis
Intervention design and compliance	Ensure cultural acceptability and practical feasibility of the intervention	Impacts adherence and increases number of drop-outs	Adherence Withdrawal	Internal	To tailor intervention in order to increase retention and adherence	Diary methods – via Interactive voice responses or SMS
Recruitment and enrolment	Guarantee representativeness of the sample and efficient recruitment methods	Risks achieving the required sample size to detect significant effect	Selection	Internal External	To determine the best possible recruitment method to reach target population	Community mapping \| Spiral walks
Randomization and allocation	Guarantee balanced randomization	Reduces comparability between groups	Selection Confounding	Internal External	To identify contextual factors that can affect the effect of the intervention and reduce comparability between groups	Observation \| Public randomization
Participant follow-up	Minimize the number of participants leaving the study	Enables unequal loss of participants between groups which can affect causal inference	Attrition Confounding (if differential attrition between trial arms)	Internal External	To prevent or understand reasons for loss to follow-up and improve retention strategies	Diary methods, mobile-based methods such as interactive voice response on SMS / WhatsApp \| peer support for adherence
Participant follow-up	Improve adherence to the intervention	Modifies the magnitude/direction of effect	Adherence	Internal	To understand reasons for non-adherence to the intervention
Data collection	Enhance reliability and quality of data	Allows inconsistent or unreliable measurements which can affect the observed magnitude and direction of the effect	Instrument	Internal	To avoid or identify errors in the measurement and data collection process.	Co-designing measurement tools with participants \| Qualitative tool validation
Data collection	Maintain objectivity during the data collection process	Threatens the validity of the data collected and/or measured to answer the objective	Instrument Recall Social desirability	Internal	To assess and validate the process of data collection.	FGD with prompts such as flashcards or images Pair interviews and role playing
Analysis and results	Identify the mechanisms underpinning the effect of the intervention	Limits an informed discussion of the results (negative or positive)	N/A	External	To triangulate the quantitative findings and identify contextual information that may have affected the results	Participatory Analysis methods

1) Background and Setting

Challenge 1: Ensure relevance of the intervention to the context.

Interviewees suggested that qualitative methods had an important role to play in ensuring that the design and adaptation of interventions corresponds with local needs. They suggested a number of activities that could be completed either (a) before testing an intervention through a trial or (b) during the trial itself, to ensure there is fidelity and that the intervention remains relevant and acceptable. For instance, an assessment to identify the local health, social, cultural and political landscape was suggested as a means of providing important information about the need for an intervention within a particular context:

In global public health, most interventions are … delivered in quite complex settings and trying to understand the setting – how people go about their daily lives and then how that intervention is going to fit into those daily lives – is really important. I16

In the qualitative literature, direct observation is frequently mentioned as a particularly useful method for developing cultural understanding of context by enabling data to be gathered through first-hand or eyewitness experience [43]. Drawing on ethnographic techniques in anthropology, observation as a method often involves sustained immersion in the research setting for a long period [44], which is not always possible as part of formative research leading up to a trial or within the constraints of a process evaluation.

As an alternative, quicker approaches to ethnography were mentioned by interview participants, including social mapping, non-participant observation, and ‘Broad Brush Surveys’, which systematically gather data on communities in a period of 5-12 days [45]:

The point of [Broad Brush surveys] is to provide a narrow and rapid impression of the visible features of the communities and get a sense of the more invisible characteristics of the communities and how they might affect the particular research and interventions planned. I11

These rapid techniques draw on the advantages of ethnographic techniques while adhering to the limitations of a trial and the need for rapid results.

Challenge 2: Obtain buy-in from stakeholders including practitioners and policy makers.

If public health interventions undergoing trials are not informed by key stakeholders, including communities, community leaders, local policy makers and health providers at national and regional levels, they risk reduced sustainability and rejection by local actors. It is particularly important to spend time engaging communities to increase participation, recruitment and to promote ownership of the problem as well as the solution, as discussed during the interviews:

The ideal would be that we engage communities even before the trial is designed in order to decide what their priorities are and then use those priorities to develop the intervention.i18

The literature points to trials that have been stopped due to significant stakeholder ethical opposition [46]. Even interventions that are effective may not translate into improved practices or policies if they are not aligned with the current agenda of local governments or policy-makers [47]. Therefore, it is important to consider and adapt to stakeholders’ concerns early on to increase subsequent dissemination and uptake of results.

Qualitative methods mentioned by interviewees to overcome these risks and promote take-up of the intervention include in-depth interviews with policy makers, observation and document analysis. This was not always done formally, but often informally as part of the early stages of a trial:

I talk to the stakeholders who are part of the PPI [Public Patient Involvement] groups, patients, and people who are delivering the intervention. I sit in on training to see what questions arise in training sessions and what people are obviously worried about when they’re thinking about delivering the intervention. I17

The literature similarly supports the use of in-depth semi-structured interviews as a means of understanding stakeholder perspectives because of the potential to produce data on people's knowledge, understandings, interpretations, experiences and interactions [48]. With this information one can enable the design of optimal, scalable interventions that could feasibly be translated into policy.

2) Intervention design and compliance

Challenge 3: Ensure cultural acceptability and feasibility.

Interventions undergoing an RCT that are not acceptable to the population or feasible in the local context will likely result in poor adherence and drop-out of participants, affecting both the magnitude of the effect and representativeness of the final sample. Tailoring the intervention to increase its acceptability can also be challenging due to a reluctance of participants to share relevant information, social desirability bias [49] and power-dynamics between researchers and participants [50]. An effective means of accessing crucial qualitative data on the acceptability and feasibility of an intervention mentioned by interviewees is diary methods:

They [fieldworkers] kept diaries because they were the people that were going on the wards every day and noticing the culture of the wards and able to see pertinent issues about the wards. I3

Recognised in the literature as a useful technique for obtaining observational data, diaries can be both qualitative and quantitative, and are used to capture and record the details of time-sensitive and context-specific phenomena [51]. As they allow events to be recorded in their natural setting, diaries reduce the delay between an event and its recording and provide a level of privacy that should reduce potential bias [52].

Interviewees also mentioned the potential for mobile phone technology to be used to collect audio diaries through an Interactive Voice Response process where participants record brief voice clips about their activities or perspectives when prompted:

So, [interactive voice response, or Short Message Service (SMS)] has been, I would say, key in helping us to track change over time but also gives us very practical information about what we need to do in terms of orienting the intervention to be most impactful. I15

Most of the research now is done on mobile phones […] it’s probably the best way to do research, to get large samples of anything. You use mobile phones here. I20

Such methods were also highlighted by interviewees as particularly useful with geographically hard-to-reach populations, tracking change in an intervention over time, or as a means of obtaining large sample sizes for data collection.

3) Participant recruitment and enrolment

Challenge 4: Guarantee representativeness and improve recruitment.

Developing recruitment techniques that reach the required sample size, while reflecting the characteristics of the broader population, is challenging:

We’ve recently been doing work in fishing communities, here, in Uganda, where it has been very much trying to get some sense of who spends time where in terms of, if you’re going to reach them with particular interventions, where are you going to find them? I10

Qualitative methods that capitalise on local knowledge can be invaluable for understanding complex social characteristics and differences that may lead to recruitment bias. To address this, both the literature and interviewees mentioned the usefulness of mapping methods, which bring together groups or communities to ‘map’ assets, processes, or physical environments. These can be extremely useful in collecting information about resources, activities, and potential recruitment locations in a relatively short period [53]. A particularly effective technique mentioned by interviewees is mapping physical spaces and resources within a community by using transect or ‘spiral’ walk methods to record how resources are distributed:

If you don’t walk through that area, you may not realise quite what’s going on. So the idea is to walk in a circle, facilitated, trying to come into contact with as many different people from as many different parts of a settlement as possible. You walk in a spiral. I10

Spiral walks use observation methods to identify people and places of significance to the research. The walk is usually in concentric circles from the centre to the outskirts of an area and can involve interviews en route [54]. The purpose of the circles is to cover all sections of a community – a transect walk in a straight line may leave out an area where a particular group of people live or work. Spiral walks were perceived by interviewees to provide a greater understanding of social structures in the community. However, in order to maximise the potential for recruitment, mapping methods should be combined with data from those implementing the recruitment procedures to assess the fit between the understanding of the social context and the way recruitment is currently being done within the trial.

4) Randomisation and allocation of the intervention

Challenge 5: Guarantee comparability between groups

One of the main challenges of trials is to ensure balance across intervention arms, in order to ensure groups are comparable and that the difference between them can still be used to achieve causal interpretation [55]. Different underpinning cluster or individual characteristics can reduce the comparability between groups, affecting the effect size and directionality of the results in unmeasurable ways. Therefore, it is important to tease out contextual factors such as community characteristics or locations that can be stratified on during randomisation. As suggested by an interviewee:

We put these communities into different types and then allow that to be part of the randomisation process so rather than just randomising according to the number of HIV prevalence or geographical location and so on. I11

Although considering contextual factors during randomisation can be difficult to achieve, it might also save a trial from losing validity. Participant observation at early stages of the study might allow researchers to witness a community’s dynamics and observe relevant information that can be used to identify strata for randomisation.

A participatory approach to identifying strata in collaboration with communities suggested during the interviews was to do public randomisation:

You know, usually it's just the scientists going into a room and using a random number generator. But putting the clusters together and also making sure that the clusters were balanced really required the community participation because they understand the agencies involved. Then even on the randomisation inside, they helped ... we had a really great statistician who did a group facilitated exercise to come up with the randomisation. I4

Involving communities in the process of identifying strata to be used in randomisation in this way can help to ensure the randomisation process considers potential similarities and differences in groups that are not directly observable by the research team. This can be a particularly useful method when there are a relatively small number of clusters to be randomised.

5) Participant follow-up

Challenge 6: Minimise the number of subjects leaving the study. Improve adherence to treatment or intervention.

Complex health interventions often address highly challenging health problems among marginalised populations through multiple intervention components. The challenging lives of targeted populations combined with the complexity of the intervention itself often magnify the number of people leaving the study and reduce potential adherence for those who stay. To effectively gather data on loss to follow-up and adherence, an interviewee recommended monitoring participants and adapting the intervention over time, consistent with a pragmatic trial approach:

…we always explicitly said that we’re going to continually measure how our intervention is working or not working in terms of whether people like it, whether people are using it, so that we can adapt our intervention as we go. I8

Interviewees also mentioned the need to maintain constant communication with participants and the advantages of doing so through highly available methods, such as SMS or WhatsApp technologies:

Wherever the population is widely using smart phones I would say, it’s worth [collecting data] on smart phones… it’s just easy to access people that way, and you can access people who you wouldn’t otherwise be able to access… you know, very specific groups. I20

The advantages of mobile communication technologies are also emphasised in the literature for their ability to provide a closer follow-up with participants, as well as acting as an alternative data collection source [56].

6) Data collection

Challenge 7: Enhance reliability and quality of the data.

Data reliability in RCTs can be affected through several mechanisms, some related to the process and others to the intrinsic nature of the data gathered. On one hand, collecting data under real-life conditions can affect the pre-standardized process of data collection. On the other hand, trials that evaluate complex public health interventions may also rely on outcomes that are subject to social desirability bias, which compromises objectivity in the measurement of outcomes.

One means of addressing this and increasing the reliability and validity of qualitative data mentioned by interviewees is the use of community-based participatory research (CBPR) methods. CBPR collaboratively involves community stakeholders, such as community groups and organisations and members of the community, at all stages of the research process [57]. The advantages of the deep involvement of participants is the potential for participants to forget they are being researched:

If they [people] are involved in an activity with a task that's group-based, I think that they forget that they’re involved in research, and so you might get slightly more accurate insights into people’s perceptions and behaviours because they don’t feel quite so much like they’re on show. I21

The literature also emphasises the potential of CBPR methods and in particular measurement tools that are co-designed with participants to ensure content validity, construct validity and consensual validity [58]. However, iterative tool development is understood as a long-term process, which should take place over multiple sessions and consultations with stakeholder groups within the community [58].

Challenge 8: Maintain objectivity during the data collection process

Many complex interventions are aimed at changing behaviours at an individual level and collect data based on self-reports, which can hinder the collection of valid data when there are complex social structures influencing people’s behaviours. An example of this challenge was introduced in one of the interviews:

So, our quantitative data showed very high rates of handwashing, but our observations showed that the reality was quite different. People often didn’t have soap in the house or only washed their hands with water, but they actually said something else in the quantitative interview. So, it gave us a better contextual understanding of the lives of those families and the structural barriers that they faced to changing their behaviour, and also, the limitations of our quantitative data, which was useful. I12

Although many RCTs select “hard” endpoint variables to ensure the identification of effect, measuring the process remains predominantly “soft”. It is a challenge for RCTs to find strategies to collect high quality, valid process data. The susceptibility to potential biases such as social desirability or recall bias increases in complex public health interventions, and identifying the best qualitative methodology to capture issues with validity becomes imperative to maintaining the objectivity of the study.

To prevent socially desirable responses from participants, interviewees mentioned the potential for embedding well-structured activities into focus group discussions to help access more reliable data. For instance, flashcards or images-as-prompts can be used to access the immediate reactions and understandings of participants, to identify social taboos, and to understand misconceptions about the intervention that could create challenges for the RCT. This same idea is evident in literature that mentions the advantages of having participants organise or "pile sort" images to assess the importance of certain ideas or issues for a group or to gain an understanding of local priorities [59]. Asking participants to order events as they believe they would or should happen can gauge a sense of people's daily lives or processes of care-seeking. Interviewees also mentioned the potential of interviewing young people in pairs or with friends, or using role-play or song, to address shyness or to obtain valuable information from younger participants:

We’ve tried friendship pair interviews where we’ve tried to get two friends to talk things through with the idea that if they’re with somebody who they feel very comfortable with the social desirability bias might be less and the friend might have some insights into the person’s behaviour. I2

All these activities offer an opportunity to more effectively access the psychological processes underlying perceptions and opinions of participants.

Challenge 9: Identify the mechanisms underpinning the effect of the intervention.

While trials are useful for evaluating the effect of a designed intervention, they fall short in explaining the mechanisms behind an intervention’s effect. Qualitative data can help address this, as acknowledged in the MRC guidance on process evaluation [39] and discussed in an interview:

…the quantitative piece will really help us understand the scale of the impact but the qualitative we’re feeding in to understanding kind of the practices and if it worked, how it worked and why it worked and we’re definitely very interested once we have the endline [data], both quantitative and qualitative to do a lot of cross-triangulation so that those two datasets kind of speak to each other. I1

Without qualitative data, trials that show little effect of an intervention will not have an explanation of why this may have occurred or which component failed. This is especially important in trials evaluating behaviour change, in which interpretations of what has changed may differ between participants and researchers. One interviewee mentioned the use of “visual participatory analysis” to understand community perceptions of the results of the trial. This method involves engaging participants in a community-based focus group discussion about the meaning of the quantitative results. During the focus groups, the participants discuss a series of graphic representation of the study’s results. This qualitative data then provided additional data that was used to inform the interpretation of the quantitative results. Where an intervention is complex and the impacts on people's lives are likely to be wide-ranging, this method offers a potential means of eliciting different perspectives on the intervention and its effects. It moves beyond recommendations that results should be disseminated to local communities [60] by offering a means of ensuring the qualitative findings from this process are integrated with the analysis of the quantitative findings from the trial.

Our findings present a new framework for integrating qualitative methods into RCTs of complex interventions with the aim of contributing to methodological discussions about the value of diverse qualitative methods for understanding complex interventions in global health and addressing the challenges of RCTs. Similar to others, our framework is a process rather than temporal framework, but with the added value of situating the benefits of using qualitative methods against the specific challenges of RCTs. This contributes directly to calls for innovation in the use of RCT methodologies to evaluate complex health interventions and the challenges they face in maintaining internal and external validity [61].

The findings of this study point specifically to the usefulness of observational and participatory methods for trials of complex public health interventions, offering a novel contribution to the broader literature about the need for mixed methods approaches. The findings highlight the benefits of using methods such as diaries, mapping techniques, and observational spiral walks to understand the social, political, and cultural context surrounding both interventions and trials. The findings also outline the benefits of using community participation to inform key trial design decisions drawing on methods such as public randomisation and ‘community-based participatory research’ as part and parcel of trials. Finally, the findings highlight the advantages of using observation and participatory approaches, such as ‘visual participatory analysis,’ to understand the effects of an intervention. This is a radical departure from the use of endline interviews with participants or focus group discussions as part of a mixed methods process evaluation. Scholars have recognised the constraints of collecting brief qualitative data during a trial and the limited understanding of the process of the intervention this generates [62]. The methods suggested as part of this study provide a means of harnessing the rich potential of qualitative research to understand context, to subsequently ensure the intervention is the best for this context, and finally to understand its effects.

In a study of how qualitative and quantitative data are combined in RCTs, The QUAlitative Research in Trials study (QUART)), researchers found that those who take an “integrated methods approach” also see qualitative research as essential to the trial and as producing evidence related to the “real world” [3]. The QUART study therefore recommends that researchers design and implement “studies not trials,” with the outcomes of the qualitative research being “central to the team’s thinking” (p124). In practice, the implementation of ‘studies’ rather than ‘trials’ requires researchers to adopt a neutral approach to methods. This essentially means selecting the best method for the research question posed rather than making presumptions about which methods are best based on a hierarchy of evidence [63]. For example, if the question is about whether people have changed health-related beliefs, then the research team potentially has multiple methods to choose from, which may include qualitative interviews, focus group discussions, photovoice, quantitative surveys, or a combination of these methods. The selection of which method is best should be driven by identifying the method that is best for answering the specific question for a particular context [64]. This approach to method neutrality may be particularly useful for complex public health interventions with strong theoretical models. Trials that are driven by theories of change or logic models have a series of research questions that may be posed alongside each stage of the model in an enumerated fashion. In a method-neutral approach, each of these stages require the research team to identify the most appropriate methods based on the question and the context. The diverse toolbox of qualitative methods suggested as part of our Framework helps to foster such an approach.

The findings presented in this article have also shown how a diverse toolkit of qualitative methods can enable appropriate adjustments during intervention development to create more sustainable interventions [7]. It can determine the relevance of a trial to participants, hence ensuring successful recruitment and follow-up, and can also identify behaviours and biases of those recruiting participants to a trial and perceptions of participants that may explain differences in outcome [28]. Finally, qualitative research can indicate why successful interventions might not work in the real world and find the best ways to ensure findings are translated to policy by establishing how results may be received by relevant parties, enhancing the utility, effectiveness and applicability of findings from RCTs [28, 65]. Regardless of whether we continue to use standardised randomised trials or seek out more pragmatic designs, we need to move beyond the division between quantitative and qualitative methods and see public health evaluation itself as a form of mixed methods research. To accomplish this, the purpose and value of qualitative methods need to be highlighted within both trial protocols and published studies, and the integration of qualitative research needs to be considered at the earliest stages of grant proposal development [13].

Implicit in the integration of quantitative and qualitative research in the evaluation of complex public health interventions is a need for greater engagement between disciplines with strong qualitative or quantitative traditions, such as anthropology and epidemiology. This is already happening to a certain extent with interdisciplinary relationships between anthropologists and epidemiologists contributing to more nuanced understandings of human behaviour and interventions better suited for local contexts [66]. However, experiences of working collaboratively across disciplines is not without its challenges and adopting a method-neutral approach as discussed above requires an acceptance of methodology as a socially-constructed tool used to ‘interpret’ rather than ‘observe’ reality [67]. This will require changes by all research actors including scholars, universities, funders and those publishing research results.

Along similar lines, a fundamental change in the publishing bias of medical journals that favours brief, quantitative articles is needed to bring method-neutral and transdisciplinary approaches to public health evaluation research [68]. In 2016, the current bias towards publishing quantitative articles led to an open letter to the British Medical Journal calling for it to “move beyond a ‘quantitative strong, qualitative weak’ stance” and to stop “rejecting qualitative research on the grounds of low priority” [69] (p2). Both the QUART study (in interviews) and our own qualitative interviews found a culture in which qualitative research is not cited as frequently as quantitative research, and is often not published in high-impact journals [69].

We argue that it is through dialogue and recognition of the complementarity of disciplines that the expansion of successful health-related interventions will be achieved. We have contributed to this dialogue by presenting innovative ways in which qualitative research can be integrated into RCTs to improve research quality and to increase health impact. We hope this encourages researchers to enter into new studies with a broader understanding of what counts as ‘evidence’ for impact evaluations and how qualitative research can strengthen the validity and rigour of RCT research for the future.

Limitations of this study

Given the foundation of this study in expert opinions and experiences, our findings are naturally influenced by the types of complex health interventions the experts interviewed were involved in. They may not be equally applicable to the wide variety of complex interventions currently being tested using different RCT designs, including those testing systems versus drugs or devices, treatment-focused versus prevention-focused interventions, cluster-randomised versus individually-randomised designs, etc. The methods we have discussed as part of our framework are also not an exhaustive list, and we would encourage researchers to investigate alternative methods from different disciplinary perspectives that meet their specific trial needs. It is also worth mentioning that our presentation of different qualitative methods does not consider the ethical implications of using these methods, and that this would need to be assessed as part of the specific aims and objectives of the trial being undertaken.

1. Rapport F, Storey M, Porter A, Snooks H, Jones K, Peconi J, Sánchez A, Siebert S, Thorne K, Clement C: Qualitative research within trials: developing a standard operating procedure for a clinical trials unit. Trials 2013, 14(1):54.

2. O'Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Hewison J: What can qualitative research do for randomised controlled trials? A systematic mapping review. BMJ open 2013, 3(6).

3. O'Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Goode J, Hewison J: Maximising the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study--a mixed methods study. Health Technology Assessment 2014, 18(38).

4. Murtagh MJ, Thomson RG, May CR, Rapley T, Heaven BR, Graham RH, Kaner EF, Stobbart L, Eccles MP: Qualitative methods in a randomised controlled trial: the role of an integrated qualitative process evaluation in providing evidence to discontinue the intervention in one arm of a trial of a decision support tool. Quality & safety in health care 2007, 16(3):224-229.

5. Lewin S, Glenton C, Oxman AD: Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ (Clinical research ed) 2009, 339:b3496.

6. Pope C, Mays N: Qualitative research: reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ (Clinical research ed) 1995, 311(6996):42-45.

7. Jansen YJ, Foets MM, de Bont AA: The contribution of qualitative research to the development of tailor-made community-based interventions in primary care: a review. European journal of public health 2009, 20(2):220-226.

8. Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, Hardeman W, Moore L, O'Cathain A, Tinati T et al: Process evaluation in complex public health intervention studies: the need for guidance. Journal of epidemiology and community health 2014, 68(2):101-102.

9. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P: Framework for design and evaluation of complex interventions to improve health. BMJ (Clinical research ed) 2000, 321(7262):694-696.

10. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. International journal of nursing studies 2013, 50(5):587-592.

11. Power R, Langhaug L, Nyamurera T, Wilson D, Bassett M, Cowan F: Developing complex interventions for rigorous evaluation—a case study from rural Zimbabwe. Health education research 2004, 19(5):570-575.

12. Petticrew M: When are complex interventions ‘complex’? When are simple interventions ‘simple’? European Journal of Public Health 2011, 21(4):397-398.

13. Drabble SJ, O’Cathain A, Thomas KJ, Rudolph A, Hewison J: Describing qualitative research undertaken with randomised controlled trials in grant proposals: a documentary analysis. BMC medical research methodology 2014, 14(1):24.

14. Ravallion M: Should the randomistas rule? The Economists' Voice 2009, 6(2).

15. Mowat R, Subramanian SV, Kawachi I: Randomized controlled trials and evidence-based policy: A multidisciplinary dialogue. Social science & medicine (1982) 2018, 210:1.

16. Elford J, Bolding G, Sherr L: Peer education has no significant impact on HIV risk behaviours among gay men in London. AIDS (London, England) 2001, 15(4):535-538.

17. Kinsman J, Nakiyingi J, Kamali A, Carpenter L, Quigley M, Pool R, Whitworth J: Evaluation of a comprehensive school-based AIDS education programme in rural Masaka, Uganda. Health education research 2001, 16(1):85-100.

18. Wight D, Raab GM, Henderson M, Abraham C, Buston K, Hart G, Scott S: Limits of teacher delivered sex education: interim behavioural outcomes from randomised trial. BMJ (Clinical research ed) 2002, 324(7351):1430.

19. Peter Craig PD, Sally Macintyre, Susan Michie, Irwin Nazarath, Mark Petticrew: Developing and evaluating complex interventions: new guidance. In. London: Medical Research Council; 2019: 39.

20. Montgomery C: From standardization to adaptation: Clinical trials and the moral economy of anticipation. Science as Culture 2016, 26(2):232-254.

21. Schwartz DL, J.: Explanatory and pragmatic attitudes in therapeutical trials. Journal of clinical epidemiology 2009, 62(5):499-505.

22. Hawe P, Shiell A, Riley T: Complex interventions: how “out of control” can a randomised controlled trial be? BMJ: British Medical Journal 2004, 328(7455):1561.

23. Hotopf M: The pragmatic randomised controlled trial. Advances in Psychiatric Treatment 2002, 8(5):326-333.

24. Chalmers I, Glasziou P: Avoidable waste in the production and reporting of research evidence. The Lancet 2009, 374(9683):86-89.

25. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E: Reducing waste from incomplete or unusable reports of biomedical research. The Lancet 2014, 383(9913):267-276.

26. Yordanov Y, Dechartres A, Porcher R, Boutron I, Altman DG, Ravaud P: Avoidable waste of research related to inadequate methods in clinical trials. BMJ (Clinical research ed) 2015, 350:h809.

27. Sandelowski M: Using qualitative methods in intervention studies. Research in nursing & health 1996, 19(4):359-364.

28. Flemming K, Adamson J, Atkin K: Improving the effectiveness of interventions in palliative care: the potential role of qualitative research in enhancing evidence from randomized controlled trials. Palliative medicine 2008, 22(2):123-131.

29. Bradley F, Wiles R, Kinmonth A-L, Mant D, Gantley M: Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP). BMJ (Clinical research ed) 1999, 318(7185):711-715.

30. Bogner A: The paradox of participation experiments. Science, Technology, & Human Values 2012, 37(5):506-527.

31. Wiles R, Crow G, Pain H: Innovation in qualitative research methods: A narrative review. Qualitative research 2011, 11(5):587-604.

32. Okoli C, Pawlowski SD: The Delphi method as a research tool: an example, design considerations and applications. Information & management 2004, 42(1):15-29.

33. Rowe G, Wright G: The Delphi technique: Past, present, and future prospects—Introduction to the special issue. Technological Forecasting and Social Change 2011, 78(9):1487-1490.

34. Gale NK, Heath G, Cameron E, Rashid S, Redwood S: Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC medical research methodology 2013, 13:117.

35. Schultz KF: CONSORT statement: updated guidelines for reporting parallel group randomised trials. BMC medicine 2010, 8:18-27.

36. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux P, Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. International Journal of Surgery 2012, 10(1):28-55.

37. Puffer S, Torgerson D, Watson J: Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ (Clinical research ed) 2003, 327(7418):785-789.

38. Datta J, Petticrew M: Challenges to evaluating complex interventions: a content analysis of published papers. BMC public health 2013, 13:568.

39. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D et al: Process evaluation of complex interventions: Medical Research Council guidance. BMJ (Clinical research ed) 2015, 350:h1258.

40. Campbell MJ: Challenges of cluster randomized trials. J Comp Eff Res 2014, 3(3):271-281.

41. Ivers NM, Taljaard M, Dixon S, Bennett C, McRae A, Taleban J, Skea Z, Brehaut JC, Boruch RF, Eccles MP et al: Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000-8. BMJ (Clinical research ed) 2011, 343:d5886.

42. Giraudeau B, Ravaud P: Preventing bias in cluster randomised trials. PLoS medicine 2009, 6(5):e1000065.

43. Harvey SA: Observe Before You Leap: Why Observation Provides Critical Insights for Formative Research and Intervention Design That You'll Never Get From Focus Groups, Interviews, or KAP Surveys. Global health, science and practice 2018, 6(2):299-316.

44. Mays N, Pope C: Qualitative research: observational methods in health care settings. BMJ (Clinical research ed) 1995, 311(6998):182-184.

45. Bond V, F. Ngwenya, E. Murray, N. Ngwenya, L. Viljoen, D. Gumede, B. Chiti, J. Mantantana, G. Hoddinott, P. J. Dodd, H. Ayles, M. Simwinga, S. Wallman, J. Seeley Value and limitations of Broad Brush Surveys used in Community-Randomized Trials in Southern Africa. Qualitative health research 2019, In press.

46. Newman PA, Rubincam C, Slack C, Essack Z, Chakrapani V, Chuang D-M, Tepjan S, Shunmugam M, Roungprakhon S, Logie C: Towards a science of community stakeholder engagement in biomedical HIV prevention trials: an embedded four-country case study. PloS one 2015, 10(8):e0135937.

47. Heneghan C, Goldacre B, Mahtani KR: Why clinical trial outcomes fail to translate into benefits for patients. Trials 2017, 18(1):122.

48. Lewis-Beck M, Bryman AE, Liao TF: The Sage encyclopedia of social science research methods: Sage Publications; 2003.

49. Latkin CA, Mai NVT, Ha TV, Sripaipan T, Zelaya C, Le Minh N, Morales G, Go VF: Social desirability response bias and other factors that may influence self-reports of substance use and HIV risk behaviors: a qualitative study of drug users in Vietnam. AIDS Education and Prevention 2016, 28(5):417-425.

50. Karnieli-Miller O, Strier R, Pessach L: Power relations in qualitative research. Qualitative health research 2009, 19(2):279-289.

51. Hyers LL: Diary Methods: Oxford University Press; 2018.

52. Lavrakas PJ: Encyclopedia of survey research methods: Sage Publications; 2008.

53. Tock L: Participatory appraisal: a brief introduction. Hull and East Yorkshire Participatory Appraisal Network 2001.

54. Ngwenya N, Gumede D, Shahmanesh M, McGrath N, Grant A, Seeley J: Community perceptions of the socio-economic structural context influencing HIV and TB risk, prevention and treatment in a high prevalence area in the era of antiretroviral therapy. African journal of AIDS research 2018, 17(1):72-81.

55. Esserman D, Allore HG, Travison TG: The Method of Randomization for Cluster-Randomized Trials: Challenges of Including Patients with Multiple Chronic Conditions. Int J Stat Med Res 2016, 5(1):2-7.

56. Singh G, Manjunatha N, Rao S, Shashidhara HN, Moirangthem S, Madegowda RK, Binukumar B, Varghese M: Use of mobile phone technology to improve follow-up at a community mental health clinic: A randomized control trial. Indian journal of psychological medicine 2017, 39(3):276.

57. Israel BA, Schulz AJ, Parker EA, Becker AB: Review of community-based research: assessing partnership approaches to improve public health. Annual review of public health 1998, 19(1):173-202.

58. Flicker S, Guta A, Larkin J, Flynn S, Fridkin A, Travers R, Pole JD, Layne C: Survey design from the ground up: Collaboratively creating the Toronto Teen Survey. Health promotion practice 2010, 11(1):112-122.

59. Colucci E: “Focus groups can be fun”: The use of activity-oriented questions in focus group discussions. Qualitative health research 2007, 17(10):1422-1433.

60. Fernandez CV, Kodish, E. and Weijer, C.: Informing study participants of research results: an ethical imperative. IRB: Ethics & Human Research 2003, 25(3):12-19.

61. Gillies C, Freemantle N, Grieve R, Sekhon J, Forder J: Advancing quantitative methods for the evaluation of complex interventions. NIHR Health Technology Assessment Programme 2016, 4(16):37-54.

62. Leeming D, Marshall J, Locke A: Understanding process and context in breastfeeding support interventions: The potential of qualitative research. Maternal & child nutrition 2017, 13(4):e12407.

63. Borgerson K: Valuing evidence: bias and the evidence hierarchy of evidence-based medicine. Perspect Biol Med 2009, 52(2):218-233.

64. Creswell JW: A Concise Introduction to Mixed Methods Research. Thousand Oaks, California: Sage Publications; 2015.

65. Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, Frankel S, Neal D, Hamdy F: for the ProtecT Study Group: Improving design and conduct of randomised controlled trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. BMJ (Clinical research ed) 2002, 325:766-770.

66. Trostle JA, & Sommerfeld, J. : Medical anthropology and epidemiology. Annual Review of Anthropology 1996, 25(1):253-274.

67. Béhague DP, Gonçalves, H., Victora, C. G. : Anthropology and epidemiology: learning epistemological lessons through a collaborative venture. Ciencia & saude coletiva 2008, 13(6):1701-1710.

68. Camlin CSJS: Qualitative research on community experiences in large HIV research trials: what have we learned? Journal of the International AIDS Society 2018, 21(S7):e25173.

69. Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, Boaden R, Braithwaite J, Britten N, Carnevale F: An open letter to The BMJ editors on qualitative research. BMJ (Clinical research ed) 2016, 352(i563).

Ethics approval and consent to participate: The study received ethical approval from the UCL Research Ethics Committee number 12449/001. Consent for participation in the study was provided in written form via email.

Consent for publication: The data reported in the article has been anonymised as much as possible to ensure that participants cannot be identified.

Availability of data and material: The data used and analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: The authors declare no competing interests.

Funding: The research was funded by the Academic of Medical Sciences in the UK. The funder was not involved in the design of the study, data collection, analysis, or interpretation of data.

Authors' contributions: We followed the ICMHE guidelines on authorship. KD wrote the first draft of the manuscript, completed the systematic review and conducted the interviews. KD and NM analysed the interview transcripts, and KD NM and JM (last author) produced the draft Framework. VB, CJC, TC, SJD, TH, ZH, JM, OM, DO, AP, JS, MS, ES, ES, and KT were participants in the original study; these authors all reviewed and commented on the draft Framework and manuscript, providing substantial contributions to the analysis and important revisions to the manuscript. NM and JM revised the manuscript based on two rounds of feedback from the participant/authors. The final version of the manuscript was read and approved by all authors. As the senior author on the paper, JM (last author) acquired the funding, and supervised the research group involved in the study.

Acknowledgements: We would like to acknowledge the insightful contributions made to the manuscript by Professor Alicia O’Cathain and Dr Laura Sheard.

Download PDF