We employed a case study for the purposes of illustrating how the critical incident technique may be applied in realist methods. This study examined the contextual conditions for team interventions in acute hospital contexts and more specifically, the enablers and barriers to team intervention success. It built on previous work which involved a systematic search of the literature using realist synthesis (19).
For the purpose of this research, a multi -disciplinary team intervention was defined as:
An intervention where a team of two or more disciplines is trying to improve how the team delivers patient care- for example: quality improvement, service improvement or change initiatives; process re-design or team training events.
These interventions were considered as complex social interventions (2) and realist evaluation was therefore considered an appropriate methodology having already been used in similar studies (12,20,21). The process commenced with a systematic search of the literature which was driven by the researcher’s own experiential knowledge and assumptions (19). Five plausible hypotheses were extrapolated using realist synthesis and are presented in the form of Context, Mechanism, and Outcome Configurations (CMOCs). Please refer to Table 2 (19).
Using these hypotheses as a foundation for the programme theory, during interviews with hospital staff who had been involved in team interventions, the authors sought to explore the conditions in which these interventions were introduced (Contexts- C); how the resources on offer in these particular contexts permeated into the reasoning of those involved in the team intervention (Mechanisms-M) and the intended and un-intended consequences of the intervention (Outcomes- O).
Please refer to Fig 1
Difficulties for the realist researcher in conducting research have already been cited (20). In this case study, the busyness of the acute hospital had potential to further impact the fieldwork process. Scheduling interviews during daily routines meant hospital staff had to consciously shift their mind-set from clinical or operational activity to the more reflective mode required for research interviews.
Prior to consideration of the CIT, trial interviews using a semi-structured format had been piloted with two purposively sampled hospital staff -one female hospital operations manager and one female hospital therapist both of whom had led on team interventions. These semi structured interviews included open ended questions for example:
“Tell me about an intervention that you have been involved in” and “How did the team operate?”
The research team reviewed the data that emerged from these interviews and agreed that significant portions of the narrative consisted of tangential generalities about teams rather than specific information related to the intervention. Participants found it difficult to construe the intervention, speaking about the team in an abstract and sometimes detached way, as evidenced in one response:
“I am not even sure of who was involved in that one…some members were only pulled in when they were needed to solve that piece of the puzzle…” [Semi-structured interview 1]
Participants demonstrated poor recall of specifics about the team or the context in which they were operating focussing instead on the process, problem or issue for which the intervention was designed. In addition, significant tangential information was collected relating to individual work patterns, relationships and practices which had limited relevance for programme theory refinement:
“Some days are cruel, you know, especially when I am in two different places…I came in this morning at half seven and I haven’t had my lunch yet…I’ve a clinic after this I need to get to, if Name} is on, I am snookered …” [Semi Structured Interview 2]
Use of this semi-structured approach had not extracted the necessary detail to meaningfully contribute to programme theory development (i.e., there were insufficient data relevant to context and mechanisms) and the research team agreed that a different format was required.
Researchers agreed that It would be necessary to elicit participant experiences before, during, and after a team intervention/programme was implemented in order to explore contextual conditions for programme theory building (22). In order to increase the quality and value of the data collected, the researchers therefore considered use of Flanagan’s Critical Incident Technique (CIT) (9) framework.
To ensure rigour in this approach, the research team first mapped and compared the five CIT procedures against the characteristics and features of realist methods (Tables 3a & 3b).
Fifteen participants who had been involved in team interventions were purposively sampled by either the Chief Operating Officer or General Manager of each participating organisation to reflect a range of disciplines; gender balance and healthcare experience across four acute hospitals in one Irish Hospital Group. Demographic information on participants is presented in Table 4.
Participants were invited to participate in the process by e-mail correspondence one week in advance of the interviews. The e-mail correspondence included an information sheet and consent form. Participants were advised that participation was voluntary and that their responses would be confidential.
Prior to the interviews being conducted, the researchers considered the application of each of the CIT procedures to the format of the interview.
CIT Procedure 1- Development of plans and specifications
The research team clarified the purpose of the critical incident interview technique (as relevant to the case study described) and agreed how to unpick the relevance of the team intervention that the participant was describing. The objective of the CIT interviews was to obtain information relevant to: team descriptors; contextual conditions (C); the objective of the intervention; outcomes (O). Probes were included with regard to how and why an intervention worked in order to elicit the mechanisms enacted (M). The interview guide was designed to give specific attention to the five CMOCs already synthesised from the literature whilst also allowing other contextual enablers and barriers to emerge and the subsequent mechanisms and outcomes generated by those conditions.
CIT procedure 2-Determination of the general aim of the activity
The research team agreed that incidents were “critical” if participants deemed them to be significant in terms of their experience and if they could be relied on as relatively accurate accounts of specific events. A “critical incident” for the purpose of this study was defined as:
“a team intervention recalled by the participant as either a significant positive or negative experience that meets research criteria in terms of being a multi-disciplinary teamintervention”.
The team considered that incidents meeting this definition were more likely to meaningfully contribute to building the IPT.
As per the CIT, interviews commenced with an introductory statement to advise participants of the purpose of the exercise (Please refer to Appendix 1. p.1 for complete protocol). Following some background questions regarding professional roles and experience, participants were asked to recall a critical incident as follows:
- Can you think of a significant event/situation/time that you were particularly proud of working on a team intervention or initiative to improve patient care?
- In a few words can you tell me what was the primary aim of the initiative was?
If recalled incidents were not deemed to meet the research criteria, participants were re-directed for the purpose of the exercise, for example:
“That’s a really nice example of an intervention introduced with your own professional colleagues, I am going to ask you to think again….this time if you can think of an intervention where there were a number of disciplines involved, that would be great.”
Following an initial question as to why they selected this experience, they were then asked a series of questions with probes embedded to elicit more factual data specific to the intervention experience. This process was followed by asking participants to recall a significant event /situation/time that they were particularly proud of, and subsequently, one there were not so proud of (See Appendix 1 Interview).
CIT procedure 3 -Collecting the data
Tests of critical incident interview format
As with the semi-structured format, the primary researcher (UC) pilot tested the new critical incident interview format on two purposefully chosen hospital staff who had been involved in leading team interventions. Following this, minor changes were made. For example, additional prompts were included to ensure interviewers probed for detail with regard to the existing CMOCs by including prompts for these in a different colour. As per the CIT, it was agreed that interviewer remarks should be neutral and permissive (23) and should demonstrate that the interviewee was the expert. However, if specifics were not emerging, clarifications could be sought for example:
“So what you are saying is….?” or “Can you give me more detail on that?”
Similarly, if information was ambiguous, interviewers could say…
“I am not sure I understood that point, am I correct in saying….?”
Following the critical incident interview trials, additional probes for data to confirm, refute or refine theories that had worked well were considered and researchers agreed on the final format for the interview (see Appendix1 supplementary materials for final interview guide).
Interviews were conducted over a period from May to September, 2018 by two members of the research team (UC & ADB). Interviews were audio-recorded with participants’ consent and transcribed verbatim. One participant did not consent to audio recording and therefore notes were taken by the interviewer during the interview process.
CIT Procedure 4 Analysing the data
Interviews were transcribed, anonymised and imported into NVivo software for the purpose of storage, analysis and interrogation of the data. (24). NVivo memo and annotation functions were used to document thought processes and decision making thus allowing the iterative process of theory building to be captured. This helped to ensure transparency (25–27) thus adhering to RAMESES II reporting standards as well as allowing for the requisite objectivity demanded by the CIT.
A critical incident was the unit of analysis (i.e., each incident described by the participant was one data unit, N= 29 incidents, as one participant could not recall a negative experience. Data were subsequently analysed in three phases summarised in Table 5 and detailed thereafter under phase 1, phase 2 and phase 3 headings. Please refer to table 5.
As recommended in the Rameses II Quality Standards for Realist Evaluation (28), a retroductive approach was adopted for analysis of the data. Retroduction refers to the movement between inductive and deductive processes to explain how and why things work the way they do. Realist researchers seek to explain the hidden causal powers of an intervention in the context in which it is applied. (29). Realist principles of retroduction stress the need for iteration within theory refinement (25). Going back and forth using both a deductive and an inductive lens enables comparison across units of analysis. The retroductive process used was consistent with the inductive lens required for CIT and also allowed existing CMOCs to be tested using a deductive lens.
Please refer to Table 6 for overview of consultations sessions.
* An interdisciplinary group of researchers and academics with a specific interest in, and experience in applying, in realist methods
Phase 1 data analysis: Scanning transcripts- Induction and Deduction
A piece of narrative was annotated if it was judged to be a relevant observation relating to the theory; for example, if it demonstrated a moderating function or appeared to refute, support or confirm prior findings. Where there was evidence of a new contextual enabler or barrier emerging, a memo was written to document how and why it was perceived to be so and to record the rationale for decisions made.
In parallel to the coding process in NVivo, a programme theory template was also developed in hard copy with each phase of the analysis colour coded to demonstrate the evolving theories and/or new emerging theories (Available on request from primary researcher UC). This was done because the primary researcher (UC) had a personal preference for reviewing the theories with the research team in hard copy as the evolution of theory refinement was considered easier to view and be interrogated at a glance.
Phase 2 Data Analysis: Building and refining theories- Retroduction
Data that were coded under the 5 original CMOCs were re-analysed and re-coded against 3 child nodes: support/ refute/ refine to allow for transparency of the process.
Narratives coded under “New CMOC” in Phase 1 were re-analysed under eight emerging theories. How and why they resulted in an intended or un-intended outcome was queried. During this process, evidence to support, refute or refine the enabling condition was first extrapolated. For transparency of the process, each of the eight emerging CMOCs were used as parent nodes and narrative was coded if there was evidence specific to Context, Mechanism and Outcome.
Phase 3 analysis
As part of the iterative process of data analysis, additional notes were made if there was evidence of moderating influences, rival mechanisms and inter-dependencies. Where refinement of the theory appeared to be indicated, an annotation was made in NVivo as to how and why the judgment for same was made. Where a judgment call could not be made by the primary researcher, a memo link was created for discussion with co- authors and the realist support group.
Both groups suggested further exploration specific to one possible new theory - “in the moment learning”. Two additional interviews were therefore undertaken in December 2018 with purposively sampled participants who had specific expertise in delivering team interventions using event simulation as a team training intervention. Two additional positive experiences of team interventions (N= 2 incidents) were analysed following the same three phase analysis (Total N= 31 incidents). Following this iterative process of data analysis, the research team agreed and finalised the initial programme theory.
CIT procedure 5- Interpreting and reporting the data.
As per the CIT, it is “imperative” that interpretation and reporting of data “is objective”(23). For this purpose, RAMESES II for realist evaluation were followed (29). In addition, the consolidated criteria for reporting qualitative studies (COREQ)was adhered to (30) . In order to understand and agree the underpinning cause of the outcomes observed, data were presented to the research team and the realist support group on two separate occasions so that the chains of inferences (CMOCs) made by the primary researcher could be challenged. This helped to maintain objectivity and rigour in the process of developing insights.