Twenty-six of the 37 NCORP sites (70%) were involved in the primary study and participants were selected from 15 of these sites. Five investigators conducted a total of 33 interviews in seven rounds from February 1, 2019 through September 3, 2020. Participants included physicians, nurses, pharmacists and other healthcare professionals. Almost all reported that they were fairly or very aware of CPGs. Demographic characteristics of participants and their institutions are provided in Table 1.
Table 1
Characteristics of participants and their institutions
Characteristic
|
Value (N = 33)
|
Participant Characteristics
|
Female Sex, n (%)
|
27 (82%)
|
Profession, n (%)
Physician
Nurse
Nurse Practitioner
Pharmacist
Othera
|
11 (33%)
11 (33%)
4 (12%)
3 (9%)
4 (12%)
|
Median years since completion of most recent training (IQR)
|
13 (6.5–22.5)
|
Median years of pediatric oncology experience (IQR)
|
13 (6.0-24.5)
|
Median years at current institution (IQR)
|
11.4 (3–17)
|
Median percentage of time spent providing direct patient care (IQR)
|
80 (50–95)
|
Self-assessed awareness of CPGs, n (%)
Very Aware
Fairly Aware
Not very aware
Missing
|
9 (27%)
21 (64%)
2 (6%)
1 (3%)
|
Site Characteristics
|
Site Location, n (%)
Western US
Southwestern US
Northeastern US
Southeastern US
Midwestern US
|
9 (27%)
7 (21%)
5 (15%)
6 (18%)
6 (18%)
|
Site Type, n (%)
Minority/Underserved Community (vs. Other)
Pediatric (vs. Mixed Adult and Pediatric)
Private (vs. Academic)
|
15 (45%)
20 (61%)
13 (39%)
|
aChild life specialist, physical therapist, psychologist |
Abbreviations: IQR – interquartile range; CPG – clinical practice guideline; US – United States |
Seven rounds of interviews were conducted in total. The first round was closed after three interviews because feedback on the initial format required immediate revision (e.g., changing paragraphs of text to bulleted lists). Overall, we found that major changes were made to the format for both the strong and weak recommendations up to round 3 but there were few changes made in the subsequent rounds. In addition, during the first three rounds, we asked about understanding of the format based on a clinical scenario for an individual patient, but in subsequent rounds, we asked participants to respond from the perspective of their institution to encourage the participant to focus on the general application of the recommendation. By the sixth round of interviews, the strong recommendation format met criteria for both understanding and saturation. Therefore, only the weak/conditional format was discussed in the seventh round of interviews. Understanding of the recommendations by participants in each round of interviews is presented in Table 2. Examples of the final recommendation formats are presented in Fig. 1 for the topic of chemotherapy-induced nausea and vomiting
Table 2
Number of participants out of 5* with good understanding of the recommendation
|
Round
1*
|
Round
2
|
Round
3
|
Round
4
|
Round
5
|
Round
6
|
Round
7
|
Weak recommendation
|
3
|
2
|
2
|
5
|
2
|
1
|
5**
|
Strong recommendation
|
1
|
5
|
4
|
4
|
5
|
5
|
-
|
* In Round 1, only 3 participants were interviewed |
** Strong recommendation format was not tested in Round 7. |
Factors influencing understanding of the recommendations
Usability: Throughout the interview a tension was noted between providing too much or too little information. During the first rounds, it became clear that interviewees found the provided information difficult to read. One participant stated, “because it’s too wordy I wouldn’t know where to start and would have to read every line.” Most of the information was initially provided in paragraphs of text. Therefore, the format was changed to bulleted lists. Conversely, many participants wanted more and specific information about drugs that were recommended, such as brand and generic names, contraindications, and dosages. To address these conflicting requests, hyperlinks to additional information were embedded within the recommendations. However, additional information provided in the rationale section was well-liked and was, therefore retained: “I like the rationale – like in general – physicians like to see that rationale.” The positioning of the rationale on the left and then more details about the evidence on the right also made sense to interviewees.
In the early rounds of interviews, interviewees commented on the specific format of the recommendations. Most comments were related to emphasizing key elements in the text, such as bolding the strength of the recommendation and the words used to denote the strength (e.g., ‘suggest’ or ‘use’), and changing the title of the document to include the strength of the recommendation. Participants also stated that strong and weak/conditional recommendations should be presented in the same general format – a ‘consistent’ presentation.
Understanding the strength of recommendations: The recommendations were first described in the formats as strong or weak. Strong recommendations are generally understood as meaning that the recommendation should be followed and were correctly interpreted by the majority of participants even in the early rounds of interviews (Table 2). The correct interpretation of the weak recommendations, however, was less clear to participants. The word weak was interpreted as “no better than a random recommendation” or “a failure” or “less important”. Participants reported that they would be less likely to follow a weak recommendation.
In consecutive rounds of interviews, we employed multiple strategies to communicate the meaning of a weak recommendation more clearly. First, we included a general statement describing the weak recommendation as “A weak recommendation will apply to the majority of patients, but may depend on circumstances, or patient or society values.” Unfortunately, this statement did not improve the understandability of the weak recommendation. Next, different symbols – a filled circle for strong, and a faded circle for weak – were used to distinguish strong from weak. Most participants indicated that the symbols simply looked like bullet points or were confusing or unnecessary and they were therefore eliminated. Then, different words were tried to convey the obligation to follow the recommendation. For example, strong recommendations had been written as ‘we recommend’ using active verbs, such as use and had been well understood. Weak recommendations were often written as ‘we suggest’ but interviewees were still confused about the intention of the recommendation. Finally, after six rounds of testing, we changed the word weak to conditional and revised the general statements describing strong and conditional recommendations. At that point, participants correctly interpreted a conditional recommendation and appropriately conveyed the intent: “conditional just sounds more legitimate, and the word conditional is great as it is very neutral. You can use it for positive or negative.” Participants also remarked that conditional did not sound “inferior” and conveys, as is intended, the need for more thought when making decisions.
Validity - Need for justification and rationale: The initial format of the recommendation included a “Rationale” section, where we described both the evidence and the justification for the recommendation. Most participants indicated that the Rationale Section was very useful and helpful to understand why the recommendation was made. Participants appreciated the information provided about the number of studies or participants in the studies, the results, and the limitations of the evidence, as it showed that the recommendation was evidence-based. This section also included an overall rating of the evidence according to the GRADE approach, which was well-liked.
At times, however, participants had additional questions related to the evidence which was not provided in detail in the rationale section. Participants questioned why a weak/conditional recommendation was based on high quality evidence, or why a strong recommendation was based on moderate quality evidence. This juxtaposition was confusing. Some participants were also confused when the evidence was available but described as low quality. One participant stated “that is the part that doesn't make sense to me. It is stating that it is weak and low quality but there is still a good amount of information… that is accurate information.” Others did not understand how evidence from research in adults could still be high quality when applied to children: “It’s hard because the data isn’t based on children so how can it be strong? But the evidence quality is high.” This suggested that more justification was needed to explain why evidence was rated at a specific quality level (e.g., because of limitations or risk of bias of the included studies, or the indirectness or applicability of the evidence).
Participants also suggested that the rationale could be communicated to a patient/patient’s family to explain why a medication or approach was being offered. In addition, they indicated that this section would be more useful if it described how institutional differences have or could have an impact on the recommendation. Since participants identified multiple roles for the Rationale Section, it was divided into two sections in the third round to better meet these needs: Supporting Evidence and Rationale. The Rationale section describes the balance of benefits, and then harms, and provides information about additional factors, such as implementation, that were considered by the CPG panel when making the recommendation. The Supporting Evidence section explains the level of evidence with hyperlinks to supporting studies when applicable, and the quality of the evidence. Key words are hyperlinked to explanatory or supplementary information.
Applicability - Acceptance and individualisation of a recommendation in practice: Similar to comments about needing a rationale (in particular for weak/conditional recommendations), participants noted that they wanted more information when a recommendation entailed an institutional practice change. Some acknowledged that a recommendation would be more easily accepted if it matched current practice: “this recommendation is standard practice, so I would do it.” Other factors related to acceptance were trust in the group who developed or endorsed the CPG and the level of supporting evidence. Even though the level of evidence is not correlated with the strength of the recommendation (e.g., a strong recommendation can be based on very low certainty evidence when there is the potential for serious harms), participants indicated that they would find it harder to accept a recommendation if it were based on low or very low-quality evidence. In addition, a participant stated that “…if it supports what I’m already doing, it doesn’t matter so much what the level of evidence is.”
To improve implementation of a recommendation, many participants requested more details, including information about the specific population to which the recommendation pertains and the circumstances. Suggestions included clearly indicating the population to which the recommendation applied in the recommendation statement and in the rationale; providing links to websites with additional descriptions or information, such as about drug doses; and providing brand names of the drugs mentioned. These changes were made after each round and improved understanding of the recommendation. Interviewees also expressed a desire for a care pathway or specific instructions, most notably when recommendations offered a choice (such as between one of several drugs). In later rounds, we changed the perspective by asking participants what should be the institutional standard of care based on the recommendation, i.e., using the recommendation to create care pathways which would apply to an individual patient. Still, there were some physician participants who indicated that they may not follow recommendations when deciding how to care for their own patients.
Presentation of the recommendation - Final format: After six rounds, 28 interviews, the final format for a strong recommendation was reached whereas seven rounds and 33 interviews were required before the format for a conditional recommendation was finalised (Fig. 1). These formats are presented as a single recommendation when printed or as single recommendations with hyperlinks to additional information when presented electronically. The recommendation strength is clearly indicated in the title and, for clarity, the meaning of the strength of the recommendation follows just below. The recommendation is clearly stated and highlighted. The rationale is featured on the left side and the supporting evidence on the right side. Under each section, information that specifically addresses the findings that arose from the interviews is presented as a bulleted list.