A total of 17 interviews were completed (8 female, 9 male) lasting from 15 to 60 minutes. Seven participants had between 1–10 years of practice experience and the remaining ten participants had more than 20 years in practice. The majority (n = 10, 58.8%) worked as part of a multidisciplinary family health team.
Participants appreciated certain design elements such as the targeted use of colour and emphasis on the number of eligible patients for a specific action as they facilitated review and interpretation of the data. Despite these changes, family physicians described challenges with identifying actions to take in response to the data which undermined the overall utility of the report. Factors that impacted the perceived usability of the report can be summarized within two key themes: (1) alignment between report and recipients’ expectations and (2) capacity to engage with QI.
Theme 1: Alignment between report and recipients’s expectations impacts usability
Family physicians described their expectations for the feedback report related to the quality indicators and data presented. First, they expected the report and it’s indicators to reflect best practice. Second, they expected the quality indicators to be valid and accurate. Third, family physicians expected the report to offer feedback on clinical activities that they perceived to be within their power to change (i.e., their control). Lastly, family physicians expected the goal of the report to be aligned with their perspectives of quality improvement. When these expectations are not met, the perceived usability of the report was low.
Subtheme 1.1: Quality indicators must reflect best practice
Family physicians expected the quality indicators to reflect best practice, which for them meant alignment with: the purpose of primary care, the clinical guidelines, their perceptions of best practice and of clinical priorities, and the realities of clinical practice. Physicians described a disconnect between the indicators and this definition of best practice, and as a result of this, a belief that the information lacked relevance to their practice, was not a priority, was not motivating and required no action. In contrast, when there was alignment between the indicators, and participant’s priorities and perceptions of clinical practice, they reported that the feedback “made sense”, was valuable, and even served to reinforce existing QI initiatives, thereby improving the perceived usability of the report.
You have to make sure … numbers are important, but the number has to reflect purpose. When you give a precise number for something that’s meaningless, you have precision of something which isn’t going to motivate. (P12)
I’m below average for LDL testing for diabetic patients, mostly because it looks like they’re looking at me doing annual LDL testing. Personally, I think the evidence points to not actually doing this on a routine basis. And I’m at average or I’m above average with respect to statin prescriptions for those diabetic patients. So, I think that kind of fits more of what we’re trying to get at, rather than the LDL testing. […] I think that testing LDL doesn’t necessarily help outcomes for my patients. (P06)
Subtheme 1.2. Quality indicators must be perceived as being valid and accurate
Participants wondered about the validity, the accuracy, the credibility and the integrity of the quality indicators (and then, the data). Data are perceived as valid when the physician believes it accurately reflects and measures the characteristics of and variations in their patient population. For one participant, the validity of quality indicator relies on its ability to link clinical performance (e.g., routine cancer screening) with huge patient outcomes (saves lives). Family physicians did not always trust the source of the data, believing it to be incorrect or outdated and leading them to trust their general perceptions over objective numbers. When the data are perceived as not useful, this negatively affects physician buy-in.
The last line that goes over the demographics is really interesting. I seem to have more of the older-age, geriatric practice and it’s kind of nice to see that because I think that influences referrals and how many times people go to the Emerg as opposed to practices that may have a much younger population. So it’s really nice that I think it acknowledges the demographic of your practice. (P11)
It gives a whole bunch of people that are not up to date with haemoglobin A1c testing, but it’s incorrect data. It says that most of our diabetics, I think our line is 13%, which is incorrect. So, all this stuff is not useful for me. (P8)
Knowing that the data is not accurate, because it’s based on [public databases] (…) I have less buy-in that the data actually reflects my real practice. Simply because there is no way for me to feed back to the system, either through [this report or others], to say that on this particular patient on this data point you don’t have it right. (P9)
Subtheme 1.3 : Quality indicators are expected to be actionable and within physicians’ control
When reviewing feedback in their report, physicians interpreted their current performance to be reflective of either action or inaction on their part or that of their patients. Family physicians expected the report to offer feedback on clinical activities they perceived to be within their control to change. When family physicians determined an indicator within the report reflected activity beyond their control, they determined the indicator was irrelevant to their practice and did not expect to see improvement over time. Being aware and in agreement of an area of practice requiring improvement can prime action:
I think it’s in my mind more. For instance, the retinal testing I was slightly below so it was just on my mind when I’m doing my diabetic checks… It primes me to do that… (P3)
One major limitation of the quality indicators was the inability to capture the shared-decision making process and the person-centered approach. A physician can offer guidance and direction but it is ultimately the patient who takes action, either completing a test or receiving a treatment. Physicians expressed some frustration because the indicators were not reflective of this shared responsibility:
I get a little irritated [… ] I mean, if you’re doing everything you can, it’s a little frustrating, because you wonder what you can do more. With these numbers, with the A1C, I see most of my diabetics every three months, so I’m thinking, well, why is that going down? Also, with the retinal scan. I mean, you have to ask them if they go to the eye doctor and they say, yes, but clearly, according to this, it’s going below the average, which means … It can be good, but it’s also frustrating, because a lot of times this is stuff out of your control. (…)I think we have to adapt our indicator to remember that people will make their own decisions and we don’t have control. (P14)
I don’t think that there are many things in my control to change those numbers and so going on again and again has felt kind of like a waste of time because I’m quite sure that nothing will be different…(P16)
Subtheme 1.4: Alignment of the goal of the report with how physicians approach quality improvements
Finally, family physicians were unclear as to goal of the report and expressed a need for clearer direction or an explicit target to support action. They expected the report to be aligned with their perspective of QI: supporting point-of-care decisions by identifying areas of improvement, offering clear guidance on how to improve performance and identifying specific targets in line with desired actions. The report was not perceived as a means of evaluating physicians’ performance because the data is not “good enough” to support this type of evaluation.
I think the question I have...is what you would like physicians in general to do with the report? Because it’s all nice to give people information but if there is no clear direction about what they should do with it and how they could integrate it easily into their day-to-day use of their EMR [electronic medical record] or of their function in the office. (P9)
So if you’re using [data] as a guide to help physicians improve their practice that’s one thing, but if you’re using it to evaluate physicians, I think the data is just not good enough for that. (P7)
However, family physicians appreciated the opportunity to see change, specifically improvement in their performance following concrete efforts to improve:
There’s great cancer screening, for sure, in terms of seeing where I’m at with that, seeing if, we do invest quite a bit of our staff time and energy into calling and mailing patients and reminding of that stuff. And so, to see that that’s paying off and that we’re not doing all that work and still below average or something. That’s very validating (P5)
Theme 2: Capacity to engage with QI impacts usability
Even when family physicians agreed that reviewing their performance data is an important part of their professional role, they described several barriers to engaging with the report. System-level conditions (e.g., time, resources) as well as work-related conditions (e.g., workload, competing priorities) impacted different stages of the QI process including accessing the data, interpreting the data and action planning.
2.1 Hard to fit A&F into workflow and resources constraints
Competing priorities were a reality for family physicians: they had heavy workload of clinical tasks each day. They also had to navigate through different duties and roles as educators, leaders, and/or managers. Some of them reported balancing their time between preventing and treating diseases which influenced what activities they prioritized. They had to weigh carefully how additional QI processes in response to A&F might fit into their workflow. Physicians highlighted that accessing their data, which means searching their patient records, is a time-consuming process which was hard to integrate into existing workflow.
We balance prevention with everything else that we do… if we followed all the good evidence in terms of prevention, and not just the things that… are in these reports… Those people, if we do what the evidence says we should do for prevention in the top ten chronic diseases, there is no time to do all the other stuff. We have to be reasonable about how we put our efforts. We could get these indicators up a lot higher, but people would be dying. It’s good that we are doing this. I’m not saying there is anything wrong with that. But the context is, this is only a tiny part of what we do. You have to look at your resources. (P13)
It just is one less step because if I see that I have 27 patients not tested for diabetes, I have to dangle into my EMR and do the search myself. So it’s extra searching and busy day it might not become the top of my list. But if it’s right there for me then I’m going to be more likely to follow up on that. (P3)
2.2 Insufficient skills to interpret data
Physicians also struggled to interpret some aspects of the report and questioned its meaning. They lacked the proper guidance and skills to do so and suggested that a co-intervention (i.e., discussing with someone they trust) would be helpful.
I’m just not sure how to interpret it. We’ll say, for example, total Emergency room visits. It tells me my practice, unadjusted, is 810 visits per 1,000 patients. Then, in the next column over, it does a risk adjustment and downgrades it to 504. I presume what that means, but I’m not entirely clear, is that my practice may be more complicated or have more comorbidities, so my number actually isn’t as bad as 810, that it’s gone down to 504 to account for that. But, again, I’m curious about that. I don’t know, does that mean I can take away from that, that I have a more complicated practice than average? (P1)a
2.3 Lack of guidance on how to prompt actions
Regarding action planning, physicians perceived the report as unactionable because i) it was not perfectly up-to-date, and ii) the aggregated nature of the data could not easily be translated into clinical actions without appropriate support. These challenges were not at all influenced by the visual nature of the report re-design.
You need to be able to see how you’re doing on the big scope of things, yeah, but you need to be able to thin it down to the individual patients that make up the bigger picture. That’s what spurs the action, to identify who they are (P2)
I think it’s nice to see the trend but at the same time how do we act on it now? And that’s what kind of deterred me from moving forward and using it more often. So I think our EMR would … and when we do a search we actually shoot out here are the patients who are overdue and then our nurses and team try to call those patients or keep it in the back of our minds. I think the summary is super nice to look at, out of interest, but again it’s not helping at a patient-specific level… (P11)