Comparing Traditional and Rapid Approaches
Comparing Time and Transcription Costs
The traditional approach required more time than the rapid approach and included transcription costs. Cohort A, using the traditional approach, required 683 total hours and $7,250 in transcription costs. Cohort B, using the rapid approach, required 409.5 total hours with no transcription costs. In effect, the rapid approach required 273.5 fewer total hours and saved $7,250 in transcription costs. However, reductions in analyst hours varied by project activity, with the greatest reductions achieved in the facility-level analysis phase. The following sections provide a summary of analyst hours and transcription costs for both approaches. See Table 1, Table 4, and Fig. 1 for additional description.
Table 4
Traditional versus Rapid Approach: Differences in Analyst Hours and Transcription Costs
Hours
|
Traditional Approach (Cohort A)
|
Rapid Approach (Cohort B)
|
Differences in Hours
|
*Total Data
|
50 interview audio hours across 16 facilities
|
0 hours
|
Data Management
|
34 total hours
1 hour/project set-up = 1
.5 hours x 50 interviews = 25
.5 hours x 16 facilities = 8
|
.5 total hours
.5 hours/project set-up = .5
0 hours x 50 interviews = 0
0 hours x 16 facilities = 0
|
33.5 hours
|
Data Collection
|
50 total hours
|
50 total hours
|
0 hours
|
Data Analysis: Interviews
|
275 total hours
5.5 hours x 50 interviews
|
196 total hours
3.92 × 50 interviews
|
79 hours
|
Data Analysis: Facilities
|
224 total hours
14 hours x 16 facilities
|
63 hours
3.92 hours x 16 facilities
|
161 hours
|
Data Interpretation
|
100 total hours
|
100 total hours
|
0 hours
|
Total Hours
|
683 hours
|
409.5 hours
|
273.5 hours
|
Transcription Cost
|
Traditional Approach (Cohort A)
|
Rapid Approach (Cohort B)
|
Differences in Cost
|
Transcription
|
$7,250
145/hour x 50 hours
|
$0
|
$7,250
|
*Cohort A included 57 interviews across 17 facilities (1–4 interviews/facility); because one facility only had one interview, the need to aggregate data for that facility was eliminated. In effect, these calculations use 16 facilities for both cohorts. Cohort B included 72 interviews across 16 facilities (3–6 interviews/facility). However, due to a higher proportion of 30-minute interviews for Cohort B, both cohorts had approximately 50 audio hours. |
Data Management
Data management in the traditional approach required 1 hour to set-up the project and .5 hours/interview plus .5 hours/facility. In contrast, data management in the rapid approach required only .5 hours to set-up the project with no other time needed. As shown in Table 1, the rapid approach eliminated data management steps except for creating the MS Excel CFIR construct by facility template. As a result, the rapid approach reduced time by 33.5 hours. Though not directly impacting analyst hours, transcripts were not received for 2–6 weeks following interviews, significantly delaying analysis for the traditional approach. See Table 1, Table 4, and Fig. 1.
Data Collection: Semi-Structured Interviews
Data collection methods were the same across both approaches and the total number of audio hours was roughly equivalent between Cohort A and B; in effect, there were not significant differences in analyst hours between approaches. However, the rapid approach required blocking approximately 3 hours for each interview: approximately 1 hour for the interview plus 1–2 hours to process the notes and “code” them into the CFIR construct by facility matrix immediately following the interview. The analyst’s immediate recall of the interview helped bolster accuracy of the notes but intensified effort and cognitive load on interview days.
Data Analysis: Traditional and Rapid Approaches
Data analysis in the traditional approach required 5.5 hours/interview plus 14 hours/facility versus 3.92 hours/interview plus 3.92 hours/facility in the rapid approach. In effect, the rapid approach reduced time by 79 hours (275 versus 196 for traditional and rapid, respectively). The largest contributor to this reduction in analyst hours was in the facility-level analysis phase; where the rapid approach required 63 hours, the traditional approach required 224 hours. See Table 1, Table 4, and Fig. 1.
Data Interpretation: Facility and Construct Analyses
Data interpretation methods were the same across both approaches, which consisted reviewing the CFIR construct by facility matrix. Both approaches took approximately 100 hours for data interpretation. See Table 1, Table 4, and Fig. 1.
Comparing Effectiveness and Rigor
There were substantial differences in the number of hours and transcription costs between the traditional and rapid approach; however, both approaches were systematic and there was concordance among many of the evaluation phases (See Table 1). Although data was condensed earlier in the rapid approach than the traditional approach, i.e., following the interview versus following the facility memo, the depth of the data in the final matrices was similar for both approaches. For example, both matrices included brief direct quotes from participants. As a result, both approaches were effective in meeting our overall goal for the evaluation; we were able to identify and describe the factors influencing implementation in a high level of detail. However, the rapid approach also allowed us to share formal results more quickly with our operational partners (See Table 5).
Table 5
Traditional versus Rapid Approach: Effectiveness and Rigor
Domain
|
Traditional Approach
|
Rapid
Approach
|
Effectiveness: Evaluation Objectives
|
|
|
Ability to identify and describe implementation determinants
|
Yes
|
Yes
|
Ability to provide rapid feedback to operational partners
|
No (preliminary results only)
|
Yes
|
Rigor: Evaluation Processes
|
|
|
Credibility
|
Analyst authority: We had analysts with expertise in both qualitative methods and the CFIR
|
Yes
|
Yes
|
Data accuracy: We used two analysts/interview and maintained access to the raw data in order to verify accuracy of data, especially quotations
|
Yes (transcripts & audio recordings)
|
Yes (audio recordings)
|
Data organization: We used matrices, allowing us to parse out and synthesize data as needed
|
Yes
|
Yes
|
Dependability
|
Data comparability: We used the same interviewers and semi-structured interview guide (based on the CFIR) to ensure data was comparable across participants and facilities
|
Yes
|
Yes
|
Analysis audit trail: We documented keys phases of analysis and edits in memos and/or matrices
|
Yes
|
Yes
|
Confirmability
|
Data triangulation: We interviewed multiple participants at each site, allowing us to triangulate data
|
Yes
|
Yes
|
Team reflexivity: We held weekly meetings to discuss discrepancies and refinements to coding processes
|
Yes
|
Yes
|
In addition, both approaches included processes to enhance methodological rigor [25, 26]. Credibility of results, a form of rigor, was most relevant when assessing tradeoffs between our rapid and traditional approaches [26]. We enhanced credibility of results by having analysts with expertise in qualitative methods and the CFIR. To ensure participant responses were accurately captured in our summaries, we used two analysts per interview as a quality check and verified summaries with raw data (transcripts or audio recordings). Overall, the final summaries from both approaches were quite similar. See Table 5 for additional description of effectiveness and concordance of rigor between both approaches.