Lipstick on a pig: Understanding efforts to redesign audit and feedback reports for primary care.

doi:10.21203/rs.3.rs-1350094/v1

Download PDF

Research

Lipstick on a pig: Understanding efforts to redesign audit and feedback reports for primary care.

https://doi.org/10.21203/rs.3.rs-1350094/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: In Ontario, Canada, a government agency known as Ontario Health is responsible for making audit and feedback reports available to all family physicians. The confidential report provides summary data on three key areas of practice: safe prescribing, cancer screening, and diabetes management. This report was redesigned to improve its usability and the objective of this study was to explore how the redesign was perceived.

Methods: We conducted qualitative semi-structured interviews with family physicians who had experience with both versions of the report, recruited through purposeful and snowball sampling. We analysed the transcripts following an emergent and iterative approach.

Results: Saturation was reached after 17 family physicians participated. Two key themes emerged as factors that impacted the perceived usability of the report: (1) alignment between report and recipients’ expectations and (2) capacity to engage in quality improvement. Family physicians expected the report and its quality indicators to reflect best practice, to be valid and accurate. They also expected the report to offer feedback on clinical activities they perceived were within their control to change. Further, family physicians expected the goal of the report to be aligned with their perspective on feasible quality improvement activities. Most of these expectations were not met, limiting the perceived usability of the report. The capacity to engage with audit and feedback was hindered by several organizational and physician-level barriers including the lack of fit with existing workflow, competing priorities, time constraints, and insufficient skills for bridging the gaps between their data and the corresponding desired actions.

Conclusions: Overall, the redesigned feedback report was not perceived as highly usable, given the misalignement between report and family physicians’ expectations as well as limited capacity to engage with the report. Consequently, the potential impact on clinical practice may be limited. Co-interventions to address the barriers of using audit and feedback report, as well as creating space for bridging together audit and feedback designers and recipients are avenues to consider for improving usability and effectiveness such quality improvement initiatives.

Audit and feedback

family physicians

primary care

qualitative

user-centered methodology

Many factors influence audit and feedback (A&F): recipients’ characteristics, targeted behaviour, A&F design itself and context.
Family physicians perceived that the A&F reports were not reflecting their practice; the quality indicators were inaccurate and not actionable.
Capacity to engage with A&F was hindered by mislignment with existing workflow, competing priorities, time constraints, and insufficient skills.
The efforts of redesigning A&F reports to improve usability were insufficient to drive practice changes in response to the data.
Our study reinforced the critical aspect of deploying co-interventions for bridging the gaps between the data and the desired actions, such as capacity building initiatives.

Audit and feedback (A&F) is a quality improvement (QI) intervention that involves the collection and analysis of population- or practice-level data (audit) and the provision and delivery of clinical performance summaries (feedback) [1, 2]. A&F is widely used across healthcare settings [3–5] by a variety of stakeholders, both to increase accountability and to improve quality of care [6]. A wide range of behaviours may be targetted including but not limited to laboratory testing and transfusion ordering [3], adherence to clinical guidelines, and prescribing [7]. Many factors influence A&F effectiveness, including the characteristics of targeted behaviour, the characteristics of the recipients (e.g., their skills and capabilities), the A&F itself (e.g., feedback display and delivery), and the context [2, 8–10]. Some targeted behaviours and contexts may be more amenable to A&F. However, all health professionals have the potential to benefit from A&F, underscoring the need to better understand whether and how to align the nature of the A&F itself with the characteristics of recipients.

Modifiable A&F-related factors include the frequency of the feedback, the format in which it is delivered (verbal, written, or both), the use of visual display, who provides the feedback (e.g., a supervisor or colleague), and the provision of explicit goals and action plans [8, 10, 11]. Best practice guidance also speaks to design elements, including the need to conceptually link summary messages with the visual display, presenting feedback in different ways, and minimizing extraneous cognitive load for recpients [1]. Aligning these features with recipient characteristics in primary care (e.g., patient specific sortable list, patient specific treatment history represented with graphs) can help recipients identify gaps in their practice and drive practice change [12]. Despite this, little attention has been paid to how A&F recipients perceive individual design elements and their impact on interaction with the feedback itself.

To address this gap, we undertook a qualitative evaluation of the redesigned MyPractice Primary Care Report in comparison to the original report in partnership with Ontario Health (OH, formerly Health Quality Ontario). The MyPractice Primary Care Report includes confidential ‘practice profiles’ developed for Ontario family physicians which provides summary data on three key areas of practice: safe prescribing, cancer screening, and diabetes management. The initial objective of this work was to evaluate whether and how the redesign improved the usability and perceived effectiveness of the report. Early interviews challenged our underlying assumption that recipients were meaningfully engaging with the original report. We then shifted our objective to exploring the perceived usability of the report in general, with an emphasis on recipients’ understanding of the report and their engagement with it.

Study design

We conducted qualitative semi-structured interviews to understand how family physicians perceived and interacted with the redesigned A&F report. The project received ethics approval from the Women’s College Hospital Research Ethics Board. We used the Consolidated Criteria for Reporting Qualitative Studies (COREQ) [13] for reporting the qualitative process (Additional file 1).

Context and setting

OH is the provincial advisor on quality in health care. It reports to Ontarians about the quality of the health care system and supports ways to improve health care quality throughout the system. OH offers a range of resources to support health care professionals provide better care, including providing doctors with information about how their practices compare with those of other doctors via the MyPractice Primary Care report. In Ontario, primary care is delivered mainly by family physicians. The provincial health insurance plan, funded by the government of Ontario, pays for all doctor visits, tests, and prescription medications measured in the MyPractice Primary Care report.

Intervention – MyPractice Primary Care Report

The MyPractice Primary Care Report was developed jointly by OH and ICES (a research institute that specializes in the analysis of health administrative data), in partnership with the Association of Family Health Teams of Ontario, Association of Ontario Health Centres, and the Ontario College of Family Physicians. The confidential and customized report provides family physicians and family health teams with confidential information about their own practice, in comparison to peers across the province.

At the time of the study, administrative data sources were used to assess a series of quality indicators: the report captured safe prescribing (e.g., opioid and benzodiazepine prescription rates), cancer screening (e.g., percentage of patients with up-to-date cancer screening tests for cervical, breast, and colon cancer), diabetes management (e.g., percentage of patients with diabetes who had two or more HbA1c tests within the past 12 months, who had diabetes and were over age 65 and had an active statin prescription, and who had a retinopathy screening test within the prior year), as well as health service utilization (e.g., emergency departments visits, hospital admissions and readmissions (by condition). It also provided clinical (chronic disease) and demographic (age, income) data on the patient population. Aggregate-level data was presented for each of the indicators, covering the prior 12 months of clinical practice. Practice improvement ideas specific to each of the topics were included to support recipients in taking action.

Family physicians in Ontario must sign-up to receive the report. Initially, the reports were designed without formal user-testing. The report was redesigned based on expert input in January 2017, using user testing approach, with the aim of improving usability. Changes included improved graphic design, a visual summary of performance compared to peers on the quality indicators, and an attempt to more clearly connect the aggregated quality indicators to suggested actions for improvement. Versions of the original and redesigned report can be found in Additional file 2. At the time of the study, to access the report, family physicians had to login to a password protected website. Starting in May 2017, the report was emailed to the participating family physicians as a PDF attachment.

Participants

Eligible participants were Ontario family physicians who were registered to receive the redesigned report following its release in May 2017 and had experience with the original version. These individuals were contacted via email by OH and invited to participate in a one-time interview with a member of the research team. A $100 honorarium in the form of a gift card was offered. Recruitment continued until data saturation was reached. A convenience sampling approach was used whereby an email outlining the study was sent to those who registered to receive the report. Those who received the email were also encouraged to share the study information with their colleagues who have also viewed the MyPractice report (snowball sampling).

Data collection

Interviews were conducted over the phone by a member of the research team (CR). The interview explored impressions of the MyPractice report, perceived usefulness of the report components, possible actions inspired by the report, and suggestions to improve the report (see Additional file 3 for interview guide). Interviews were audio-recorded and transcribed verbatim by an independent third party.

One-on-one, semi-structured interviews were conducted over the telephone by a member of the research team (CR). Consent was obtained verbally by the interviewer. While viewing their confidential report, participants were asked about their overall impressions of the report, whether they felt the report was easy to navigate and what parts of the report they found most useful and why. Participants were also asked to describe what action(s) (if any) they took following their review of the report (e.g., conducted a chart review to determine who amoung their patients with diabetes were due for a HbA1c test), and what feature(s) of the report informed those actions.

Data analysis

The qualitative analysis followed an emergent and iterative approach. We initially informed our codebook following the Brehaut et al. suggestions for best practice in A&F [1]. Three members of the research team (CR, BB, NK) independently coded a first transcript. They then met to compare interpretation of targeted quotes and revise the codebook. All interviews were double-coded. Peer debriefing was conducted: preliminary findings were discussed in a meeting with senior investigators (LD, NMI) who have both conducted multiple prior qualitative studies involving A&F. The team concluded that the data were pointing to broader questions related to the perceived design and usability of the reports. At this stage, the team re-analyzed the data taking a more inductive lens. LD coded two transcripts to get immersed in the data. The focus of the data analysis shifted to an emphasis on recipients’ understanding of the A&F reports and their engagement with them. A conventional content analysis was performed following an inductive approach, meaning that codes were derived from the data [14]. Codes and categories were iteratively revised throughout this second stage of data analysis to answer the study objective. CR then met with another member of the research team (LD) to discuss and refine the categorization of codes and establish themes. Three members of the research team (CR, LD, NI) met to further refine themes, which were then finalized by all authors. CR maintained a consistent audit trail of the codebook throughout the four stages of data analysis. The study team met during the coding (initial phase) and during the regroupement of codes into categories.

A total of 17 interviews were completed (8 female, 9 male) lasting from 15 to 60 minutes. Seven participants had between 1–10 years of practice experience and the remaining ten participants had more than 20 years in practice. The majority (n = 10, 58.8%) worked as part of a multidisciplinary family health team.

Participants appreciated certain design elements such as the targeted use of colour and emphasis on the number of eligible patients for a specific action as they facilitated review and interpretation of the data. Despite these changes, family physicians described challenges with identifying actions to take in response to the data which undermined the overall utility of the report. Factors that impacted the perceived usability of the report can be summarized within two key themes: (1) alignment between report and recipients’ expectations and (2) capacity to engage with QI.

Theme 1: Alignment between report and recipients’s expectations impacts usability

Family physicians described their expectations for the feedback report related to the quality indicators and data presented. First, they expected the report and it’s indicators to reflect best practice. Second, they expected the quality indicators to be valid and accurate. Third, family physicians expected the report to offer feedback on clinical activities that they perceived to be within their power to change (i.e., their control). Lastly, family physicians expected the goal of the report to be aligned with their perspectives of quality improvement. When these expectations are not met, the perceived usability of the report was low.

Subtheme 1.1: Quality indicators must reflect best practice

Family physicians expected the quality indicators to reflect best practice, which for them meant alignment with: the purpose of primary care, the clinical guidelines, their perceptions of best practice and of clinical priorities, and the realities of clinical practice. Physicians described a disconnect between the indicators and this definition of best practice, and as a result of this, a belief that the information lacked relevance to their practice, was not a priority, was not motivating and required no action. In contrast, when there was alignment between the indicators, and participant’s priorities and perceptions of clinical practice, they reported that the feedback “made sense”, was valuable, and even served to reinforce existing QI initiatives, thereby improving the perceived usability of the report.

You have to make sure … numbers are important, but the number has to reflect purpose. When you give a precise number for something that’s meaningless, you have precision of something which isn’t going to motivate. (P12)

I’m below average for LDL testing for diabetic patients, mostly because it looks like they’re looking at me doing annual LDL testing. Personally, I think the evidence points to not actually doing this on a routine basis. And I’m at average or I’m above average with respect to statin prescriptions for those diabetic patients. So, I think that kind of fits more of what we’re trying to get at, rather than the LDL testing. […] I think that testing LDL doesn’t necessarily help outcomes for my patients. (P06)

Subtheme 1.2. Quality indicators must be perceived as being valid and accurate

Participants wondered about the validity, the accuracy, the credibility and the integrity of the quality indicators (and then, the data). Data are perceived as valid when the physician believes it accurately reflects and measures the characteristics of and variations in their patient population. For one participant, the validity of quality indicator relies on its ability to link clinical performance (e.g., routine cancer screening) with huge patient outcomes (saves lives). Family physicians did not always trust the source of the data, believing it to be incorrect or outdated and leading them to trust their general perceptions over objective numbers. When the data are perceived as not useful, this negatively affects physician buy-in.

The last line that goes over the demographics is really interesting. I seem to have more of the older-age, geriatric practice and it’s kind of nice to see that because I think that influences referrals and how many times people go to the Emerg as opposed to practices that may have a much younger population. So it’s really nice that I think it acknowledges the demographic of your practice. (P11)

It gives a whole bunch of people that are not up to date with haemoglobin A1c testing, but it’s incorrect data. It says that most of our diabetics, I think our line is 13%, which is incorrect. So, all this stuff is not useful for me. (P8)

Knowing that the data is not accurate, because it’s based on [public databases] (…) I have less buy-in that the data actually reflects my real practice. Simply because there is no way for me to feed back to the system, either through [this report or others], to say that on this particular patient on this data point you don’t have it right. (P9)

Subtheme 1.3 : Quality indicators are expected to be actionable and within physicians’ control

When reviewing feedback in their report, physicians interpreted their current performance to be reflective of either action or inaction on their part or that of their patients. Family physicians expected the report to offer feedback on clinical activities they perceived to be within their control to change. When family physicians determined an indicator within the report reflected activity beyond their control, they determined the indicator was irrelevant to their practice and did not expect to see improvement over time. Being aware and in agreement of an area of practice requiring improvement can prime action:

I think it’s in my mind more. For instance, the retinal testing I was slightly below so it was just on my mind when I’m doing my diabetic checks… It primes me to do that… (P3)

One major limitation of the quality indicators was the inability to capture the shared-decision making process and the person-centered approach. A physician can offer guidance and direction but it is ultimately the patient who takes action, either completing a test or receiving a treatment. Physicians expressed some frustration because the indicators were not reflective of this shared responsibility:

I get a little irritated [… ] I mean, if you’re doing everything you can, it’s a little frustrating, because you wonder what you can do more. With these numbers, with the A1C, I see most of my diabetics every three months, so I’m thinking, well, why is that going down? Also, with the retinal scan. I mean, you have to ask them if they go to the eye doctor and they say, yes, but clearly, according to this, it’s going below the average, which means … It can be good, but it’s also frustrating, because a lot of times this is stuff out of your control. (…)I think we have to adapt our indicator to remember that people will make their own decisions and we don’t have control. (P14)

I don’t think that there are many things in my control to change those numbers and so going on again and again has felt kind of like a waste of time because I’m quite sure that nothing will be different…(P16)

Subtheme 1.4: Alignment of the goal of the report with how physicians approach quality improvements

Finally, family physicians were unclear as to goal of the report and expressed a need for clearer direction or an explicit target to support action. They expected the report to be aligned with their perspective of QI: supporting point-of-care decisions by identifying areas of improvement, offering clear guidance on how to improve performance and identifying specific targets in line with desired actions. The report was not perceived as a means of evaluating physicians’ performance because the data is not “good enough” to support this type of evaluation.

I think the question I have...is what you would like physicians in general to do with the report? Because it’s all nice to give people information but if there is no clear direction about what they should do with it and how they could integrate it easily into their day-to-day use of their EMR [electronic medical record] or of their function in the office. (P9)

So if you’re using [data] as a guide to help physicians improve their practice that’s one thing, but if you’re using it to evaluate physicians, I think the data is just not good enough for that. (P7)

However, family physicians appreciated the opportunity to see change, specifically improvement in their performance following concrete efforts to improve:

There’s great cancer screening, for sure, in terms of seeing where I’m at with that, seeing if, we do invest quite a bit of our staff time and energy into calling and mailing patients and reminding of that stuff. And so, to see that that’s paying off and that we’re not doing all that work and still below average or something. That’s very validating (P5)

Theme 2: Capacity to engage with QI impacts usability

Even when family physicians agreed that reviewing their performance data is an important part of their professional role, they described several barriers to engaging with the report. System-level conditions (e.g., time, resources) as well as work-related conditions (e.g., workload, competing priorities) impacted different stages of the QI process including accessing the data, interpreting the data and action planning.

2.1 Hard to fit A&F into workflow and resources constraints

Competing priorities were a reality for family physicians: they had heavy workload of clinical tasks each day. They also had to navigate through different duties and roles as educators, leaders, and/or managers. Some of them reported balancing their time between preventing and treating diseases which influenced what activities they prioritized. They had to weigh carefully how additional QI processes in response to A&F might fit into their workflow. Physicians highlighted that accessing their data, which means searching their patient records, is a time-consuming process which was hard to integrate into existing workflow.

We balance prevention with everything else that we do… if we followed all the good evidence in terms of prevention, and not just the things that… are in these reports… Those people, if we do what the evidence says we should do for prevention in the top ten chronic diseases, there is no time to do all the other stuff. We have to be reasonable about how we put our efforts. We could get these indicators up a lot higher, but people would be dying. It’s good that we are doing this. I’m not saying there is anything wrong with that. But the context is, this is only a tiny part of what we do. You have to look at your resources. (P13)

It just is one less step because if I see that I have 27 patients not tested for diabetes, I have to dangle into my EMR and do the search myself. So it’s extra searching and busy day it might not become the top of my list. But if it’s right there for me then I’m going to be more likely to follow up on that. (P3)

2.2 Insufficient skills to interpret data

Physicians also struggled to interpret some aspects of the report and questioned its meaning. They lacked the proper guidance and skills to do so and suggested that a co-intervention (i.e., discussing with someone they trust) would be helpful.

I’m just not sure how to interpret it. We’ll say, for example, total Emergency room visits. It tells me my practice, unadjusted, is 810 visits per 1,000 patients. Then, in the next column over, it does a risk adjustment and downgrades it to 504. I presume what that means, but I’m not entirely clear, is that my practice may be more complicated or have more comorbidities, so my number actually isn’t as bad as 810, that it’s gone down to 504 to account for that. But, again, I’m curious about that. I don’t know, does that mean I can take away from that, that I have a more complicated practice than average? (P1)a

2.3 Lack of guidance on how to prompt actions

Regarding action planning, physicians perceived the report as unactionable because i) it was not perfectly up-to-date, and ii) the aggregated nature of the data could not easily be translated into clinical actions without appropriate support. These challenges were not at all influenced by the visual nature of the report re-design.

You need to be able to see how you’re doing on the big scope of things, yeah, but you need to be able to thin it down to the individual patients that make up the bigger picture. That’s what spurs the action, to identify who they are (P2)

I think it’s nice to see the trend but at the same time how do we act on it now? And that’s what kind of deterred me from moving forward and using it more often. So I think our EMR would … and when we do a search we actually shoot out here are the patients who are overdue and then our nurses and team try to call those patients or keep it in the back of our minds. I think the summary is super nice to look at, out of interest, but again it’s not helping at a patient-specific level… (P11)

The findings suggest that the redesign was not sufficient to drive practice changes in response to the data. While many of the indicators were perceived as low priority, physicians appreciated that their patients’ characteristics were reflected in the feedback report. Even if they were unconvinced that the indicators were the right targets for action, some became newly self-aware of the gaps in their practice. While this awareness may be a trigger in initiating professional behaviour change process, the desired actions for QI were unfortunately perceived as uncertain or infeasible.

The capacity to engage with MyPractice report was hindered by several organizational and physician-level barriers including the lack of fit with existing workflow, competing priorities, time constraints, and insufficient guidance and skills in how to interpret the data and in how bridging the gaps between their data and the corresponding actions. Given these limitations, the efforts at redesign, while appreciated by end-users, were akin to “putting lipstick on a pig”. Overall, our findings highlighted a mismatch between the goals of health system leaders designing feedback and of primary care clinicians’ reality receiving that same feedback.

Comparison of our findings with literature

Overarching best practices for designing and implementing A&F are available [1, 15, 16]. It is possible that these best practices should be seen as hierarchical – some may matter more than others. For example, our study shows that even if the ‘design’ features of feedback display can all be addressed (e.g., provide feedback in more than one way such as presenting key messages both textually and numerically), if the focus of the A&F is not aligned with recipients’ goals, and if audit itself is perceived as lacking validity, accuracy, credibility, poorly aligned with physicians’ priorities, or not readily actionable, then the intervention will not achieve its potential to improve quality.

Our findings build on Clinical Performance Feedback Intervention Theory (CP-FIT)[2] by expanding our understanding of how feedback, recipient and context variables influence the feedback cycle. Our findings show that physicians were unclear about the broad goal of the QI intervention which underscores the importance of aligning feedback goals and recipients’ goals and priorities [15, 16]. The importance and the relevance of feedback goal are key variables that impact on recipients’ acceptance and their intention of adopting/changing their behaviours [2]. Family physicians in this study wanted clearer direction of what to do with the report but also, clarity on the purpose and meaning of the whole A&F initiative (i.e. evaluating and measuring physicians’ performance vs improving practice). Recommendations around the nature of desired actions further specify the need for alignment with established goals and priorities[1], which may be enhanced by including an exemplar action plan that could be adopted in response to the A&F [15].

Some family physicians thought that the quality indicators did not fairly reflect their practice and attributed the data to patient behaviours (e.g., screening). In this case, physicians felt that the data represented activities beyond their control, highlighting the importance of controllability, which can impact negatively the acceptance of the report [2]. Physicians noted that best practice elements of care, specifically patient-centerdness and shared decision-making, were not reflected in their data. Consequently, physicians felt judged for their performance based on data for which they are not entirely responsible for, causing frustration. Other studies have also highlighted the need for quality indicators to reflect the important role of patient choices [6, 17] as well as measures representing the patients’ perspective on care, the clinical quality, and the general quality of care from a broader perspective [18]. Which quality indicators are included in A&F should be a central focus of the development process and can be supported through co-design and user-centred design (UCD) methods [19–21] as well as other approaches in which end-users are actively involved [22]. UCD has been applied in many settings such as in home healthcare [23], primary care [24, 25] and long-term care [26]. Cooke et al. [22, 27] illustrated a process in which A&F reports designers and physicians (end-users) collaborated to design and implement A&F. The physicians were identified key clinical questions, in making individualized A&F reports and developed a plan for change through participation in a group feedback session. By incorporating end-user feedback into the design of A&F, UCD helps to ensure that the design of A&F reports is functional, can support end-user needs and goals, and ultimately positively influence clinical practice [23, 26].

Family physicians’ views and QI knowledge and skills (or lack thereof) influence how they interact with A&F [2, 17], highlighting the need for co-interventions. Educational strategies delivered alongside A&F have been effective in supporting improved adherence to guidelines [28], reducing the rate of cesarean delivery [29] and antibiotic prescribing [30]. In these studies, strategies have been operationalized in different ways such as a one-hour group session [28], quarterly educational outreach visits conducted by external facilitators [29] as well as two sessions of voluntary continuing medical education in addition of educational materials [30]. Training-based interventions effectively build skills [31] and improve communication skills [32], while an emphasis on data interpretation and action planning is likely to positively influence practice change [2, 33].

Limitations

First, the transferability of our findings is limited, given the context, the focused sample and the sampling approach (i.e., 17 family physicians in Ontario who voluntary signed up to the A&F report, including 10 who were part of a family health team) as well as the specificity of the QI intervention (MyPractice report) examined. The way the A&F was delivered in our study was a passive and solitary approach whereby physicians accessed their reports in confidence via e-mail. A&F initiatives that support the creation of space for physicians to discuss the data with colleagues and/or a credible source and enable greater understanding and actions for improvement may be perceived as more usable. Methodologically, no member checking process was undertaken to validate data interpretation among research participants. However, we held peer debrief meetings with research team, supported by senior researchers, to review the data analysis and the findings, as well as to discuss the findings interpretation. Finally, to address the change in research objective as mentioned previously, we described the research process in a transparent way, and we went back to the data to analyse and interprete them consistently to answer the research questions.

This study found that esthetic design changes played a minor role in how family physicians used the A&F report. Usability of A&F appears to depend more on recipients perceptions of whether the quality indicators are important, accurately measured, and controllable through feasible clinical actions. Those who found the A&F report useful did so because they felt was aligned with the goals and priorities of their practice as a whole. Many family physicians would benefit from co-interventions to facilitate the integration of A&F into workflow, and to build capacity to interpret the data and undertake practice-level actions. Health system administrators and A&F report recipients should work together to optimize alignment between report, and the priorities of end-users.

A1C

blood test reflecting average blood glucose levels

A&F

audit and feedback

EMR

electronic medical record

Ontario Health

quality improvement

LDL

low-density lipoprotein UCD:user-centred design

Ethics approval and consent to participate

The project received ethics approval from the Women’s College Hospital Research Ethics Board (#2017-0083-E). Consent to participate in the study was obtained verbally by the interviewer.

Consent for publication

All authors have read and approved the final manuscript.

Availability of data and material

An aggregate summary of the data generated during this study is included in this published article. Individual data transcripts cannot be shared publicly due to confidentiality.

Competing interests

This study was funded by the Ontario SPOR SUPPORT Unit, which is supported by the Canadian Institutes of Health Research and the Province of Ontario. NI is supported by a Canada Research Chair in Implementation of Evidence-Based Practice and by the Department of Family and Community Medicine at Women's College Hospital and the University of Toronto. He holds research grants from Canadian Institutes of Health Research and the Ontario Government.

Funding

This study was funded by the Canadian Institutes for Health Research through the Ontario SPOR SUPPOR Unit. The funder was not involved in the design of the study or the writing of this manuscript.

Authors' contributions

CR and GR drafted the manuscript. GR participated in the data analysis and revised the manuscript. CR collected and analysed the data. NI led the development of the protocol, obtained funding, contributed to all aspects of the study design. LD reviewed the study design, supervised and participated in the data analysis. All authors revised substantially this manuscript and approved the submitted version and have agreed both to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the study, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Acknowledgements

The authors would like to acknowledge and thank the family physicians who participated in that project. The authors would also like to acknowledge the staff at Ontario Health Quality for their help in making this work possible. We are grateful to Holly Witteman and David Flaherty who both revised the first draft of this paper. Finally, ther authors thank the research team members who were involved in previous data analysis cycles: Natasha Kithulegoda and Beth Bosiak.

Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, et al. Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Ann Intern Med. 2016;164:435–41.
Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers N, Francis JJ, et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci. 2019;14:40.
Foster M, Presseau J, McCleary N, Carroll K, McIntyre L, Hutton B, et al. Audit and feedback to improve laboratory test and transfusion ordering in critical care: a systematic review. Implement Sci IS. 2020;15:46.
Tomsic I, Heinze NR, Chaberny IF, Krauth C, Schock B, von Lengerke T. Implementation interventions in preventing surgical site infections in abdominal surgery: a systematic review. BMC Health Serv Res. 2020;20:236.
Van Den Bulck S, Spitaels D, Vaes B, Goderis G, Hermens R, Vankrunkelsven P. The effect of electronic audits and feedback in primary care and factors that contribute to their effectiveness: a systematic review. Int J Qual Health Care. 2020;32:708–20.
Ivers N, Barnsley J, Upshur R, Tu K, Shah B, Grimshaw J, et al. “My approach to this job is … one person at a time”: Perceived discordance between population-level quality targets and patient-centred care. Can Fam Physician The College of Family Physicians of Canada. 2014;60:258–66.
Lindsay PJ, Rohailla S, Taggart LR, Lightfoot D, Havey T, Daneman N, et al. Antimicrobial Stewardship and Intensive Care Unit (ICU) Mortality: A systematic review. Clin Infect Dis Off Publ Infect Dis Soc Am. 2019;68:748–56.
Hysong SJ. Meta-Analysis. Audit & Feedback Features Impact Effectiveness on Care Quality. Med Care. 2009;47:356–63.
Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci IS. 2006;1:9.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;CD000259.
Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 1982. 2010;70:1618–25.
Guldberg TL, Vedsted P, Lauritzen T, Zoffmann V. Suboptimal quality of type 2 diabetes care discovered through electronic feedback led to increased nurse–GP cooperation. A qualitative study. Prim Care Diabetes Elsevier. 2010;4:33–9.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57.
Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qual Health Res. 2005;15:1277–88.
Jamtvedt G, Flottorp S, Ivers N. Audit and Feedback as a Quality Strategy. Improv Healthc Qual Eur Charact Eff Implement Differ Strateg [Internet]. European Observatory on Health Systems and Policies; 2019 [cited 2021 Nov 26]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK549284/.
Ivers N, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, et al. No more ’business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci IS. England: BioMed Central Ltd; 2014;9:14–14.
Desveaux L, Ivers NM, Devotta K, Ramji N, Weyman K, Kiran T. Unpacking the intention to action gap: a qualitative study understanding how physicians engage with audit and feedback. Implement Sci. 2021;16:19.
Arvidsson E, Dahlin S, Anell A. Conditions and barriers for quality improvement work: a qualitative study of how professionals and health centre managers experience audit and feedback practices in Swedish primary care. BMC Fam Pract. 2021;22:113.
Dopp AR, Parisi KE, Munson SA, Lyon AR. Aligning implementation and user-centered design strategies to enhance the impact of health services: results from a concept mapping study. Implement Sci Commun. 2020;1:17.
Colquhoun HL, Sattler D, Chan C, Walji T, Palumbo R, Chalmers I, et al. Applying User-Centered Design to Develop an Audit and Feedback Intervention for the Home Care Sector. Home Health Care Manag Pract. 29: SAGE Publications Inc; 2017. pp. 148–60.
McNamara P, Shaller D, Ivers N. Confidential Physician Feedback Reports: Designing for Optimal Impact on Performance. Rockville: Agency for Healthcare Research; 2016 p. 42. Report No.: AHRQ Pub. No. 16-0017-EF.
Cooke LJ, Duncan D, Rivera L, Dowling SK, Symonds C, Armson H. The Calgary Audit and Feedback Framework: a practical, evidence-informed approach for the design and implementation of socially constructed learning interventions using audit and group feedback. Implement Sci. 2018;13:136.
Colquhoun HL, Sattler D, Chan C, Walji T, Palumbo R, Chalmers I, et al. Applying User-Centered Design to Develop an Audit and Feedback Intervention for the Home Care Sector. Home Health Care Manag Pract. 29: SAGE Publications Inc; 2017. pp. 148–60.
Brown B, Young J, Smith DP, Kneebone AB, Brooks AJ, Egger S, et al. A multidisciplinary team-oriented intervention to increase guideline recommended care for high-risk prostate cancer: A stepped-wedge cluster randomised implementation trial. Implement Sci IS. 2018;13:43.
Brown B, Balatsoukas P, Williams R, Sperrin M, Buchan I. Multi-method laboratory user evaluation of an actionable clinical performance information system: Implications for usability and patient safety. J Biomed Inform. 2018;77:62–80.
Landis-Lewis Z, Kononowech J, Scott WJ, Hogikyan RV, Carpenter JG, Periyakoil VS, et al. Designing clinical practice feedback reports: three steps illustrated in Veterans Health Affairs long-term care facilities and programs. Implement Sci. 2020;15:7.
Cooke LJ, Duncan D, Rivera L, Dowling SK, Symonds C, Armson H. How do physicians behave when they participate in audit and feedback activities in a group with their peers? Implement Sci IS. 2018;13:104.
Lesuis N, van Vollenhoven RF, Akkermans RP, Verhoef LM, Hulscher ME, den Broeder AA. Rheumatologists’ guideline adherence in rheumatoid arthritis: a randomised controlled study on electronic decision support, education and feedback. Clin Exp Rheumatol. 2018;36:21–8.
Chaillet N, Dumont A, Abrahamowicz M, Pasquier J-C, Audibert F, Monnier P, et al. A Cluster-Randomized Trial to Reduce Cesarean Delivery Rates in Quebec. N Engl J Med Massachusetts Medical Society. 2015;372:1710–21.
Pettersson E, Vernby Å, Mölstad S, Lundborg CS. Can a multifaceted educational intervention targeting both nurses and physicians change the prescribing of antibiotics to nursing home residents? A cluster randomized controlled trial. J Antimicrob Chemother. 2011;66:2659–66.
Michie S, Atkins L, West R. The Behaviour Change Wheel Book - A Guide To Designing Interventions [Internet]. Silverback Publishing. Great Britain; 2014 [cited 2021 Sep 8]. Available from: http://www.behaviourchangewheel.com/online-book#1.
Little P, Stuart B, Francis N, Douglas E, Tonkin-Crine S, Anthierens S, et al. Effects of internet-based training on antibiotic prescribing rates for acute respiratory-tract infections: a multinational, cluster, randomised, factorial, controlled trial. Lancet Lond Engl. 2013;382:1175–82.
van der Velden AW, Kuyvenhoven MM, Verheij TJM. Improving antibiotic prescribing quality by an intervention embedded in the primary care practice accreditation: the ARTI4 randomized trial. J Antimicrob Chemother. 2016;71:257–63.

Download PDF

Version 1

posted

You are reading this latest preprint version

Lipstick on a pig: Understanding efforts to redesign audit and feedback reports for primary care.

Status:

Version 1

Abstract

Contributions To The Literature

Background

Methods

Data analysis

Results

Discussion

Conclusion

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and material

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Supplementary Files

Status:

Version 1