Prior to consideration of the CIT, trial interviews using a semi-structured interview (SSI) format had been piloted with two purposively sampled staff members -one female Operations Manager and one female Therapist both of whom had led on team interventions. These SSIs included open ended questions for example:
“Tell me about an intervention that you have been involved in” and “How did the team operate?”
The research team reviewed the data that emerged from these interviews and agreed that significant portions of the narrative consisted of tangential generalities about teams rather than specific information related to the intervention. Data were therefore not relevant to programme theory development (i.e., insufficient data relevant to context and mechanisms) and a different format was required. The CIT was subsequently considered. In order to ensure rigour, the authors mapped and compared the five CIT procedures against the characteristics and features of realist methods (Table 3).
CIT Procedure 1- Development of plans and specifications
The research team clarified the purpose of the critical incident interview technique (as relevant to the case study described) and how to unpick the relevance of the team intervention that the key informant was describing. The objective of the CIT interviews was to obtain information relevant to: team descriptors; contextual conditions (C); the objective of the intervention; outcomes (O) and would also include probes with regard to how and why an intervention worked in order to elicit the mechanisms enacted (M). The interview guide would be designed to give specific attention to the five CMOCs already synthesised from the literature whilst also allowing other contextual enablers and barriers to emerge and the subsequent mechanisms and outcomes generated by those conditions.
CIT procedure 2-Determination of the general aim of the activity
Incidents described would be considered “critical” if KIs deemed them to be significant in terms of their experience and if they could be relied on as relatively accurate accounts of specific events. A “critical incident” for the purpose of this study was defined as:
“a team intervention recalled by the KI as either a significant positive or negative experience that meets research criteria in terms of multi-disciplinary team and team intervention and therefore has potential to contribute to building the IPT”.
As per the CIT, interviews would commence with an introductory statement to advise KIs of the purpose of the exercise (Please refer to Appendix 1. p.1 for complete protocol). Following some background questions regarding professional roles and experience, KIs would then then be asked to recall a critical incident as follows:
- Can you think of a significant event/situation/time that you were particularly proud of working on a team intervention or initiative to improve patient care?
- In a few words can you tell me what was the primary aim of the initiative was?
If recalled incidents were not deemed to meet the research criteria, KIs would be re-directed for the purpose of the exercise, for example:
“That’s a really nice example of an intervention introduced with your own professional colleagues, I am going to ask you to think again….this time if you can think of an intervention where there were a number of disciplines involved, that would be great.”
Following an initial question as to why they selected this experience, they would then be asked a series of questions with probes embedded to elicit more factual data specific to the intervention experience. This process was followed in asking participants to recall a significant event /situation/time that they were particularly proud of, and subsequently, one there were not so proud of (See Appendix 1 Interview).
CIT procedure 3 -Collecting the data
The CIT interview format was piloted on two KIs by the primary researcher (UC) and minor changes were made for example interviewers were reminded where to probe for detail with regard to the existing CMOCs by including triggers for these in a different colour. As per the CIT, it was agreed that interviewer remarks should be neutral and permissive (23) and should demonstrate that the interviewee was the expert. However, if specifics were not emerging, clarifications would be sought for example:
“So what you are saying is….?” or “Can you give me more detail on that?”
Similarly, if information was ambiguous, interviewers might say…
“I am not sure I understood that point, am I correct in saying….?”
Following the pilot tests, probes for data to confirm, refute or refine theories that had worked well were discussed by all authors and agreement was reached on the final format for the interview.
Fifteen KIs who had been involved in team interventions were purposively sampled by either the Chief Operating Officer or General Manager of each participating organisation to reflect a range of disciplines; gender balance and healthcare experience across 4 acute hospitals in one Irish Hospital Group. Demograhics information of participants is presented in Table 4.
KIs were invited to participate in the process by e-mail correspondence one week in advance of the interviews. The e-mail correspondence included an information sheet (See Appendix 2} and consent form. Participants were advised that participation was voluntary and that their responses would be confidential. Interviews were conducted over a period from May-September, 2018 by two members of the research team (UC & ADB). Interviews were audio-recorded with the KIs’ consent and transcribed verbatim. One interviewee did not consent to audio recording and therefore notes were taken by the interviewer during the interview process.
CIT Procedure 4 Analysing the data
Fifteen interviews (N= 29 incidents, as one KI could not recall a negative experience and 2 interviews took place at a later stage) were transcribed, anonymised and imported into NViVO software (23) in order to manage the analysis. This was conducted in three phases and included deductive, inductive and retroductive approaches to data analysis.
See Table 5 Summary of data analysis
Data were analysed against the five existing theories (CMOCs) that had been extrapolated from the literature in the form of plausible hypotheses. Deductive logic was employed to test these 5 CMOCs in order to see whether associations matched the expectations.
Using inductive logic, narratives were also reviewed to extrapolate evidence that suggested the emergence of new theories. As programme theory development is an iterative process, the co-authors and the realist researcher methodology support group* were consulted to provide advice and feedback at each phase of the evaluation process.
Please refer to Table 6 for overview of consultation sessions.
* An interdisciplinary group of researchers and academics with a specific interest in, and experience in applying, in realist methods
Phase 1 data analysis: Scanning transcripts- Induction and Deduction
Using the 5 CMOCs that had emerged from the literature as parent nodes, phase 1 analysis involved an initial scanning of the transcripts. Pieces of narrative were coded according to parent nodes “team descriptors”; “CMOCs 1-5” or if there was the possibility of “a new CMOC” emerging.
Please refer to Table 7 –Phase 1 NVIVo coding
A piece of narrative was annotated if it was judged to be a relevant observation relating to the theory; for example, if it demonstrated a moderating function or appeared to refute, support or confirm prior findings. Where there was evidence of a new contextual enabler or barrier emerging, a memo was written to document how and why it was perceived to be and how and why that judgment call was being made.
In parallel to the coding process in NViVO, a programme theory template was also developed with each phase of the analysis colour coded to demonstrate the evolving theories and /or new emerging theories. (Available on request from primary author UC).
Phase 2 Data Analysis: Building and refining theories- Retroduction
Data that were coded under the 5 original CMOCs were re-analysed and re-coded against 3 child nodes: support/ refute/ refine to allow for transparency of the process. Please refer to Table 8 -Phase 2a NViVO coding.
Narratives coded under “New CMOC” in Phase 1 were re-analysed under 8 emerging theories. How and why they resulted in an intended or un-intended outcome was queried. During this process, evidence to support, refute or refine the enabling condition was first extrapolated. For transparency of the process, each of the 8 emerging CMOCs were used as parent nodes and narrative was coded if there was evidence specific to Context, Mechanism and Outcome.
Please refer to Table 9- Phase 2b NViVo coding
Phase 3 analysis
As part of the iterative process of data analysis, additional notes were made if there was evidence of moderating influences, rival mechanisms and inter-dependencies. Where refinement of the theory appeared to be indicated, a note was made as to how and why the judgment for same was made. Where a judgment call could not be made by the primary researcher, a note was written for discussion with co- authors and the realist support group.
Both groups suggested further exploration specific to one possible theory - “in the moment learning”. Two additional interviews were therefore undertaken in December 2018 with purposively sampled interviewees who had specific expertise in delivering team interventions using simulation. Two additional positive experiences of team interventions (N= 2 incidents) were analysed following the same three phase analysis (Total N= 31 incidents). Following this iterative process of data analysis, co-authors agreed and finalised the IPT.
CIT procedure 5- Interpreting and reporting the data.
As per the CIT, it is “imperative” that interpretation and reporting of data “is objective”(23). For this purpose, Realist and Meta-narrative, Evidence Synthesis, Evolving Standards (RAMESES) for realist evaluation were followed (24). In addition, the consolidated criteria for reporting qualitative studies (COREQ): 32-item checklist was adhered to (25) . In order to understand and agree the underpinning cause of the outcomes observed, data were presented to co-authors and the realist support group on two separate occasions so that the chains of inferences (CMOCs) made by the primary researcher could be challenged. This helped to maintain objectivity and rigour in the process of developing insights.