Study Setting and Design
This study took place in Ontario, Canada. The aim of our study was to ascribe to each of the 14 Ontario health regions a ‘KT signature score’ related to the types of KT interventions and approaches to intervention implementation used in each region to improve the quality of rectal cancer surgery. We used interviews with key stakeholders to obtain data on region-level KT interventions related to rectal cancer surgery that occurred over years 2006 to 2014. Our audit started at year 2006 since this was the year when Cancer Care Ontario became more active in attempting to influence the quality of rectal cancer surgery through KT interventions. We summarized results into narrative and visual forms. KT experts then scored these summaries using a KT Signature Assessment Tool and grouped regions with similar scores. (See Figure I –Methods Flow Diagram)
Design of Interview Guide
A 25-page interview guide helped identify if an intervention did or did not occur and processes of intervention implementation. (Additional File I) The Cochrane Effective Practice and Organization of Care taxonomy outlined an exhaustive list of KT interventions that may have been used including: education materials (e.g., guidelines), education meetings, audit and feedback, practice demonstrations, education outreach or detailing, reminders, and, tailoring interventions.15 Activities potentially provided though Cancer Care Ontario but not specifically listed in the Cochrane taxonomy such as communities of practice, diagnostic assessment programs, and multidisciplinary cancer conferences were also included. They were included in the interview guide since the goal of these activities is to improve relevant care consistent with optimal current standards, the presumed mechanism of action is the improvement of knowledge among clinicians, and, effectiveness is contingent on local clinician participation. Positive responses were probed further to understand the processes of intervention implementation. Probes considered the following: was the activity selected by an individual or group; what body did such individuals or groups represent; were interventions selected to address specific quality gaps; were interventions delivered at the individual surgeon, hospital or region level; and, how was intervention success evaluated? There was special interest in identifying surgeon-led iKT targeting region-level performance, and, evidence of sustained iterative approaches (e.g., data exercises that were repeated through time and not simply one-off evaluations). These latter concepts were considered reflective of more progressive and effective KT approaches.
The Surgical Oncology Program at Cancer Care Ontario assigns a surgical oncology lead and a colorectal cancer surgery lead for each of the 14 Ontario health regions. These leads were invited to participate under the premise that they were the most likely surgeons to be familiar with rectal cancer surgery KT initiatives in their respective region. In addition, heads of general surgery at high volume hospitals (i.e., performed >10 rectal or rectosigmoid cancer procedures per year) were approached for interviews. Snowball sampling was used to identify other key informants well positioned to provide relevant information.16 Interviewees received no compensation for participation.
Data Collection and Organisation
In advance of interviews, participants received an introductory package that included the purpose of the study and a summary of the interview guide. The summary listed pre-identified KT interventions and relevant processes of intervention selection. A single research coordinator conducted telephone interviews. Following participant consent, the interviews were recorded and transcribed verbatim.
For each region the research team summarized data on implemented KT interventions and processes of implementation into two forms. First, a narrative form summarized the following: 1) KT activities at the provincial, region and individual hospital level, with provincial efforts being common to all regions; 2) how quality gaps were identified and how interventions were selected; and, 3) the KT interventions implemented over the years in the region, with further detail on processes of implementation for each intervention. Second, a KTA cycle was populated where appropriate with specific interventions and processes. For member-checking, narrative summaries and KTA Cycles for individual regions were mailed to interview participants for review and feedback. To further establish comprehensive data gathering, respective region narrative summaries and KTA Cycles were sent to other respective regional stakeholders including Cancer Care Ontario Regional Vice Presidents, Cancer Care Ontario Surgical Oncology and Colorectal Surgery leads (to the small number of such individuals who did not participate in interviews), and Chiefs of Surgery at all hospitals in each region. Covering letters emphasized the importance of stakeholders reviewing summarized region processes and KT activities, since these would be used to assign a KT signature to each region for future quantitative analyses. Feedback and corrections were encouraged. Small clarifications were received from two regions and incorporated into final narrative summaries and KTA cycles.
Outcomes and Analysis
The primary outcome for this study was a ‘KT signature score’ ascribed to each of the 14 Ontario health regions. The intention was for such scores to reflect the breadth of and approaches to KT intervention implementation meant to improve a region’s quality of rectal cancer surgery. The study team could find no instruments to score KT intervention implementation across a large region and over an extended period of time in any clinical area; or even any articles attempting to do this using either qualitative or quantitative methods. Therefore, we devised a practical KT Signature Assessment Tool. (See Figure II) This tool listed 20 items corresponding to processes of implementation (e.g., the use of region-level data to identify quality gaps) and specific KT interventions (e.g., guidelines). Four experts in KT reviewed the tool and provided feedback on its design prior to use. For this initial attempt at assigning KT signature scores, apriori it was decided that each of the resulting 20 items would be scored on a Likert scale from 1-5, where 1 and 5 represented the item or process used ‘not at all’ or ‘to a great extent’, respectively. The maximum and minimum score for each region was therefore 100 (20 items x 5 = 100) and 20, respectively.
Assigning a KT Signature Score to Regions
The above four KT experts also participated in a modified Delphi process to assign KT signature scores to individual regions using our collected data.17,18 Experts were first provided with the study objectives and methods, a copy of the final KT Signature Assessment Tool, and, the region summaries (narrative and KTA Cycle). Face-to-face meetings were then arranged. Meetings began with a study overview, and then presentation of each region’s narrative and KTA cycle summary. Primary data were also available for direct review. Following each presentation, raters independently scored intervention implementation using the KT Signature Assessment Tool. It was decided a priori that for each region scores from raters would be averaged. Regions were thus rank-ordered according to these average scores. A priori it was also decided that regions with similar ‘KT Signature’ scores would be placed into groups for potential future quantitative analyses. Experts discussed average scores and the region summaries to formulate region groupings. Consensus for groupings was reached and then re-visited and re-confirmed through post-meeting email.