Improving Understandability of Clinical Practice Guideline Recommendations: a Qualitative Study to Develop a Format for Pediatric Cancer Care

Background: Features of clinical practice guideline (CPG) recommendations may affect understanding and consequently successful uptake and implementation. We aimed to develop a recommendation format to improve understandability among healthcare professionals involved in pediatric cancer care. Methods: We conducted a multi-center qualitative study of health care professionals at participating pediatric oncology sites. We developed an initial format based on the current literature and used the “think-aloud” technique in multiple rounds of one-on-one cognitive interviews to iteratively improve it. Interviews were conducted until the format was well understood and no new, substantive suggestions for revision were raised. We took a directed (deductive) approach to content analysis of the interview notes to identify concerns related to the understandability of the recommendation. Results: Five investigators interviewed 33 healthcare professionals from multiple disciplines in seven rounds. We identied important factors inuencing how to communicate recommendations. Regarding the strength of the recommendation, participants found understanding weak recommendations more challenging than strong. Understanding was improved by using the word ‘conditional’ instead of ‘weak’. Participants believed the inclusion of a rationale section to be very helpful. More information was desirable when a recommendation entailed a practice change. Although participants wanted additional information, they were concerned that there could be too much information. They, therefore, suggested that key words and the studies included in the evidence synthesis be hyperlinked to explanatory data. In the nal format, the recommendation strength is clearly indicated in the title, highlighted, and dened within a text box. The rationale for the recommendation is in a column on the left, with supporting evidence on the right. In a bulleted list, the rationale section describes the benets and harms and additional factors, such as implementation, that were balanced by the CPG developers. Each bullet under the supporting evidence section indicates the level of evidence with an explanation and the supporting studies with hyperlinks when applicable. Conclusions: A well-understood recommendation format to present strong and conditional recommendations was created through an iterative interview process. The format is straightforward, making it easy for organizations and CPG developers to use it to communicate recommendations clearly to intended users.

Participants believed the inclusion of a rationale section to be very helpful. More information was desirable when a recommendation entailed a practice change. Although participants wanted additional information, they were concerned that there could be too much information. They, therefore, suggested that key words and the studies included in the evidence synthesis be hyperlinked to explanatory data. In the nal format, the recommendation strength is clearly indicated in the title, highlighted, and de ned within a text box. The rationale for the recommendation is in a column on the left, with supporting evidence on the right. In a bulleted list, the rationale section describes the bene ts and harms and additional factors, such as implementation, that were balanced by the CPG developers. Each bullet under the supporting evidence section indicates the level of evidence with an explanation and the supporting studies with hyperlinks when applicable.
Conclusions: A well-understood recommendation format to present strong and conditional recommendations was created through an iterative interview process. The format is straightforward, making it easy for organizations and CPG developers to use it to communicate recommendations clearly to intended users.

Contributions To The Literature
Implementation of clinical practice guideline recommendations continues to be challenging and may be related to the inherent characteristics of recommendation formats.
In addition to the presentation of a recommendation, we showed that other factors have an important in uence on the interpretation and understanding, such as strength of the recommendation, the relationship between strength and the quality of the supporting evidence and impact on changes in practice.
Providing the rationale and the supporting evidence for the recommendation and using terms such as "conditional" and "strong" successfully communicated the intention of the recommendation.

Background
Many organizations have dedicated time and resources to develop clinical practice guidelines (CPGs), with countless hours committed by the guideline development group crafting the recommendations and supporting materials, with the goal of improving uptake and implementation. Despite these efforts, implementation has proved challenging across many professions and disciplines. While much of the literature has focused on how to implement recommendations in practice (such as through auditing and providing feedback to practitioners), there has been increasing interest in determining the inherent features of the recommendations themselves and their format that could in uence implementation. (1,2) The guideline implementability framework of Gagliardi et al., for example, outlines four speci c domains related to the presentation of recommendations that affect implementability: Usability, Applicability, Validity and Adaptability.(3) Usability refers to how the recommendation and the evidence is presented and can include how users navigate the information; applicability addresses the inclusion of contextual information to promote use of the recommendation for an individual; validity covers how the evidence is summarized and presented for ease of interpretation; and adaptability focuses on the different versions of a guideline to improve uptake.
With respect to usability, the layout and choice of wording of the recommendation, as well as the supporting evidence could in uence the successful communication of the recommendation. Some guidance exists to improve the communicate of the intended action of the recommendation. The National Academy of Science's (formerly known as the Institute of Medicine) Standards for Developing Trustworthy Clinical Practice Guidelines indicate that recommendations should clearly describe the action and the strength of the recommendation: whether the recommendation should be in the a rmative (i.e. 'to do something') or the negative (i.e. 'not do something') and how to frame the level of obligation to follow the recommendation.(4) When using the Grading of Recommendation, Assessment, Development, and Evaluation (GRADE) approach, the strength of a recommendation is classi ed as either strong or weak/conditional.(5) Strong recommendations are associated with a high level of obligation to provide (or withhold) an intervention as the standard of care. Conversely, weak/conditional recommendations prompt clinicians to also consider values, preferences or other factors when applying the recommendation.
Based on these guidance statements for recommendation development, the wording of any recommendation should convey its strength intrinsically. However, the choice of word or words to convey this strength has proven to be challenging. Investigators have evaluated 'recommend' versus 'suggest' and 'should' versus 'might' ,and found that no word or wording was superior in communicating the recommendation strength. (6) Others have examined words such as "should", "may" and "must" and found that 'must' conveyed the highest level of obligation, and "may consider" the lowest level of obligation. (7) However, there was considerable overlap in the level of obligation conveyed. It is also unclear how much information should accompany recommendations to ensure understanding and acceptance. Some research supports the need for more explanation when a recommendation is based on a close balance of the evidence, or when its implementation requires that other factors be considered. (8) These ndings suggest that more rationale and implementation information is necessary for weak/conditional recommendations to be understood as compared to strong recommendations.
We undertook this qualitative study to develop a format to communicate recommendations to health care professionals, speci cally in pediatric oncology. This multi-center, qualitative study is one of three substudies within a Children's Oncology Group (COG) study (ACCL15N1CD). The overall goal is to investigate the use of CPGs at pediatric National Cancer Institute (NCI) Community Oncology Research Program (NCORP) sites by understanding current practice in supportive care, exploring the barriers and facilitators to CPG uptake, and nally, developing a format to present recommendations to improve understanding and uptake.

Methods
We conducted one-on-one interviews to develop a recommendation format that conveys all appropriate information to pediatric oncology healthcare professionals. We used the "Think Aloud Technique" (TAL) of cognitive interviewing in multiple rounds to iteratively improve the format by exploring participants' understanding of the recommendations and preferences for the presentation. We applied a directed content analysis to explore multiple themes from within the interviews. We used the Standards for Reporting Qualitative Research (9) to guide the report of our ndings.

Participants
All 37 COG-member NCORP sites were invited to participate in the primary study. We invited healthcare professionals who provided direct care for children with cancer at participating sites to be interviewed. Trainees were excluded. Potential participants submitted their demographic characteristics and were purposively sampled to ensure variability by type of healthcare professional, years in pediatric oncology practice, self-professed level of familiarity with CPGs (from not very to very aware), NCORP site location, and proportion of time devoted to direct patient care. Participants gave verbal consent to be interviewed; the need for documentation of consent was waived by the NCI Pediatric Central Institutional Review Board.

Development of the recommendation format
The development of the initial recommendation format (Additional File 1) was informed by literature and health literacy principles. We considered the quantity of text (e.g., brief versus more detailed), the type of information (e.g., rationale and summary of evidence), formatting (e.g., use of bulleted lists, font size, typeface, text boxes), use of symbols, and use of color. From recommendations available from COGendorsed supportive care CPGs,(10) we chose three topics (fever and neutropenia; chemotherapy-induced nausea and vomiting; and platelet transfusions), and applied the format to a weak/conditional and a strong recommendation from each topic, for a total of six examples.

Interviews
We conducted one-on-one, semi-structured interviews via video chat (utilizing Go-to-Meeting™) lasting approximately 30 to 60 minutes. Five members of the team conducted interviews (MB, LLD, PDR, AMW, NS) who had experience in health research methodology and guideline development, and/or paediatric oncology. Each session was recorded, and interviewers took notes during and after the interview -all without identifying the interviewee.
Each participant was randomly assigned a topic (fever and neutropenia; chemotherapy-induced nausea and vomiting; or platelet transfusions) and shown two recommendations on that topic (1 weak/conditional and 1 strong) following a pre-determined random order. At the beginning of the interview, we described a short clinical scenario related to the recommendation topic to place the following recommendations in context. Using the TAL technique, we asked each participant for their initial reactions to the recommendation, and then continually prompted the participant to talk aloud while he/she interpreted each section of the recommendation. To evaluate understanding, we asked the participant to choose from four possible courses of action suggested by the recommendation: to provide care (or not) as per the recommendation (strong recommendations) or likely provide care (or not) as per the recommendation. (weak or conditional recommendations). Next, the interviewer disclosed the correct course of action according to the interview guide and asked the participant to re ect upon which parts of the recommendation enhanced or detracted from the correct understanding, and to suggest modi cations to improve understanding. The interview guide is available in Additional File 2.
The interviews were conducted in rounds of ve, but each round could be stopped prior to completion of ve interviews if feedback indicated that immediate revision to the recommendation format was needed.
Interviews were conducted until the recommendation format tested was well understood (de ned as correctly choosing the action recommended) by at least four of ve interviewees, and until saturation of ideas (de ned as receipt of no new, substantive suggestions) was achieved. We planned to interview 25 participants at minimum and estimated that up to 50 interviews could be required.

Data analysis
Feedback about the format was categorised as positive (not requiring a modi cation), obstructive (preventing use of the recommendation), major (interviewee able to solve independently), minor (cosmetic), and suggestions for modi cation/improvement. In addition, we calculated the number of participants per round who correctly interpreted the course of action for the recommendation. The team met virtually after each round of interviews to review the interview notes and agreed by consensus about which revisions should be made. The recommendation format was then revised prior to the next round of interviews. In order to identify other concerns related to recommendation understandability, two investigators (NS and LLD) conducted a directed content analysis of the notes taken during and after each interview based on the implementability framework. (3) To ensure trustworthiness and credibility, the analysis was reviewed and revised by the other members of the interviewing team (MB, PDR, AMW).

Results
Twenty-six of the 37 NCORP sites (70%) were involved in the primary study and participants were selected from 15 of these sites. Five investigators conducted a total of 33 interviews in seven rounds from February 1, 2019 through September 3, 2020. Participants included physicians, nurses, pharmacists and other healthcare professionals. Almost all reported that they were fairly or very aware of CPGs. Demographic characteristics of participants and their institutions are provided in Table 1. Seven rounds of interviews were conducted in total. The rst round was closed after three interviews because feedback on the initial format required immediate revision (e.g., changing paragraphs of text to bulleted lists). Overall, we found that major changes were made to the format for both the strong and weak recommendations up to round 3 but there were few changes made in the subsequent rounds. In addition, during the rst three rounds, we asked about understanding of the format based on a clinical scenario for an individual patient, but in subsequent rounds, we asked participants to respond from the perspective of their institution to encourage the participant to focus on the general application of the recommendation. By the sixth round of interviews, the strong recommendation format met criteria for both understanding and saturation. Therefore, only the weak/conditional format was discussed in the seventh round of interviews. Understanding of the recommendations by participants in each round of interviews is presented in Table 2. Examples of the nal recommendation formats are presented in Fig. 1 for the topic of chemotherapy-induced nausea and vomiting Factors in uencing understanding of the recommendations Usability: Throughout the interview a tension was noted between providing too much or too little information. During the rst rounds, it became clear that interviewees found the provided information di cult to read. One participant stated, "because it's too wordy I wouldn't know where to start and would have to read every line." Most of the information was initially provided in paragraphs of text. Therefore, the format was changed to bulleted lists. Conversely, many participants wanted more and speci c information about drugs that were recommended, such as brand and generic names, contraindications, and dosages. To address these con icting requests, hyperlinks to additional information were embedded within the recommendations. However, additional information provided in the rationale section was wellliked and was, therefore retained: "I like the rationale -like in general -physicians like to see that rationale." The positioning of the rationale on the left and then more details about the evidence on the right also made sense to interviewees.
In the early rounds of interviews, interviewees commented on the speci c format of the recommendations. Most comments were related to emphasizing key elements in the text, such as bolding the strength of the recommendation and the words used to denote the strength (e.g., 'suggest' or 'use'), and changing the title of the document to include the strength of the recommendation. Participants also stated that strong and weak/conditional recommendations should be presented in the same general format -a 'consistent' presentation.
Understanding the strength of recommendations: The recommendations were rst described in the formats as strong or weak. Strong recommendations are generally understood as meaning that the recommendation should be followed and were correctly interpreted by the majority of participants even in the early rounds of interviews ( Table 2). The correct interpretation of the weak recommendations, however, was less clear to participants. The word weak was interpreted as "no better than a random recommendation" or "a failure" or "less important". Participants reported that they would be less likely to follow a weak recommendation.
In consecutive rounds of interviews, we employed multiple strategies to communicate the meaning of a weak recommendation more clearly. First, we included a general statement describing the weak recommendation as "A weak recommendation will apply to the majority of patients, but may depend on circumstances, or patient or society values." Unfortunately, this statement did not improve the understandability of the weak recommendation. Next, different symbols -a lled circle for strong, and a faded circle for weak -were used to distinguish strong from weak. Most participants indicated that the symbols simply looked like bullet points or were confusing or unnecessary and they were therefore eliminated. Then, different words were tried to convey the obligation to follow the recommendation. For example, strong recommendations had been written as 'we recommend' using active verbs, such as use and had been well understood. Weak recommendations were often written as 'we suggest' but interviewees were still confused about the intention of the recommendation. Finally, after six rounds of testing, we changed the word weak to conditional and revised the general statements describing strong and conditional recommendations. At that point, participants correctly interpreted a conditional recommendation and appropriately conveyed the intent: "conditional just sounds more legitimate, and the word conditional is great as it is very neutral. You can use it for positive or negative." Participants also remarked that conditional did not sound "inferior" and conveys, as is intended, the need for more thought when making decisions.
Validity -Need for justi cation and rationale: The initial format of the recommendation included a "Rationale" section, where we described both the evidence and the justi cation for the recommendation. Most participants indicated that the Rationale Section was very useful and helpful to understand why the recommendation was made. Participants appreciated the information provided about the number of studies or participants in the studies, the results, and the limitations of the evidence, as it showed that the recommendation was evidence-based. This section also included an overall rating of the evidence according to the GRADE approach, which was well-liked.
At times, however, participants had additional questions related to the evidence which was not provided in detail in the rationale section. Participants questioned why a weak/conditional recommendation was based on high quality evidence, or why a strong recommendation was based on moderate quality evidence. This juxtaposition was confusing. Some participants were also confused when the evidence was available but described as low quality. One participant stated "that is the part that doesn't make sense to me. It is stating that it is weak and low quality but there is still a good amount of information… that is accurate information." Others did not understand how evidence from research in adults could still be high quality when applied to children: "It's hard because the data isn't based on children so how can it be strong? But the evidence quality is high." This suggested that more justi cation was needed to explain why evidence was rated at a speci c quality level (e.g., because of limitations or risk of bias of the included studies, or the indirectness or applicability of the evidence).
Participants also suggested that the rationale could be communicated to a patient/patient's family to explain why a medication or approach was being offered. In addition, they indicated that this section would be more useful if it described how institutional differences have or could have an impact on the recommendation. Since participants identi ed multiple roles for the Rationale Section, it was divided into two sections in the third round to better meet these needs: Supporting Evidence and Rationale. The Rationale section describes the balance of bene ts, and then harms, and provides information about additional factors, such as implementation, that were considered by the CPG panel when making the recommendation. The Supporting Evidence section explains the level of evidence with hyperlinks to supporting studies when applicable, and the quality of the evidence. Key words are hyperlinked to explanatory or supplementary information.
Applicability -Acceptance and individualisation of a recommendation in practice: Similar to comments about needing a rationale (in particular for weak/conditional recommendations), participants noted that they wanted more information when a recommendation entailed an institutional practice change. Some acknowledged that a recommendation would be more easily accepted if it matched current practice: "this recommendation is standard practice, so I would do it." Other factors related to acceptance were trust in the group who developed or endorsed the CPG and the level of supporting evidence. Even though the level of evidence is not correlated with the strength of the recommendation (e.g., a strong recommendation can be based on very low certainty evidence when there is the potential for serious harms), participants indicated that they would nd it harder to accept a recommendation if it were based on low or very lowquality evidence. In addition, a participant stated that "…if it supports what I'm already doing, it doesn't matter so much what the level of evidence is." To improve implementation of a recommendation, many participants requested more details, including information about the speci c population to which the recommendation pertains and the circumstances.
Suggestions included clearly indicating the population to which the recommendation applied in the recommendation statement and in the rationale; providing links to websites with additional descriptions or information, such as about drug doses; and providing brand names of the drugs mentioned. These changes were made after each round and improved understanding of the recommendation. Interviewees also expressed a desire for a care pathway or speci c instructions, most notably when recommendations offered a choice (such as between one of several drugs). In later rounds, we changed the perspective by asking participants what should be the institutional standard of care based on the recommendation, i.e., using the recommendation to create care pathways which would apply to an individual patient. Still, there were some physician participants who indicated that they may not follow recommendations when deciding how to care for their own patients.
Presentation of the recommendation -Final format: After six rounds, 28 interviews, the nal format for a strong recommendation was reached whereas seven rounds and 33 interviews were required before the format for a conditional recommendation was nalised (Fig. 1). These formats are presented as a single recommendation when printed or as single recommendations with hyperlinks to additional information when presented electronically. The recommendation strength is clearly indicated in the title and, for clarity, the meaning of the strength of the recommendation follows just below. The recommendation is clearly stated and highlighted. The rationale is featured on the left side and the supporting evidence on the right side. Under each section, information that speci cally addresses the ndings that arose from the interviews is presented as a bulleted list.

Discussion
Based on the results of this study of healthcare professionals in pediatric oncology, we created a format for the presentation of supportive care recommendations to improve understandability. In doing so, we obtained valuable insight into the challenges faced by users as they attempted to interpret the recommendations, and we incorporated these ndings into the nal format. One of the most striking issues that we uncovered was the misunderstanding of and discomfort with weak recommendations.
This supports anecdotal evidence that practitioners may have di culty understanding and applying recommendations that are weak, and that strong recommendations are more easily understood. Many CPG developers also prefer to write strong recommendations even when they are not warranted by the evidence or the circumstances. (11) It is possible that this preference is driven by our observed negative attitudes toward weak recommendations. Our nal recommendation format successfully addressed many of these concerns by including clear de nitions of strong and weak recommendations, changing the word weak to conditional, providing a rationale for the recommendation, and including supporting evidence and factors for users to consider when implementing conditional recommendations. We had also hypothesised that symbols might improve the understandability of recommendations, as they could be a way to distinguish the strength of the recommendation. We, however, did not nd that the use of symbols or different shading had an impact on understanding and therefore removed symbols from the format early in the process.
We tested recommendations on three different supportive care topics with different levels of evidence associated with different strengths of recommendations. We were, therefore, able to explore participants' understanding when the level of evidence did not intuitively match the strength of the recommendationfor example, a weak/conditional recommendation that was supported by moderate or high-quality evidence. These cases seemed the most confusing to the interviewees. However, by including some details about the evidence, (such as the limitation of the studies or the participants in the studies) interviewees understood the level of evidence and the strength of the recommendation. However, caution in describing the evidence is needed, as participants might use the description of the evidence to determine their own level of evidence, instead of considering the level assessed by the CPG developers.
An interesting nding of this study was the initial desire of many participants in early interviews for clinical pathways instead of recommendations. Clinical pathways are de ned as tools that standardize care over a course of treatment by translating evidence into a structured, multidisciplinary care plan within a speci c context. (12) A clinical pathway will include decisions regarding the use of available resources in a speci c setting and thus will include information about the use of speci c drugs and doses. A CPG may make a strong recommendation for the use of a drug class; a care pathway will state that a speci c member of that drug class be used. Misunderstanding of the distinction between a care pathway and a CPG may explain why many participants desired speci city in the recommendation format. It may also explain why weak/conditional recommendations were less well understood, since a care pathway may offer little room for discretion even when based on a weak/conditional recommendation. It may also be for this reason that we found that most participants did not want more information presented in the recommendation format, but preferred links to other detailed information if needed. While multi-layered approaches -where increasing amounts of information are provided in layers -have been explored by other investigators (13), we did not perceive that participants wanted another layer of explanatory information as part of the format that speci cally described the recommendation or the evidence. Nor did we nd that more than a sentence about the implementation of the recommendation was needed. Nonetheless, if more information was desired, users could seek out the complete published CPG.
One of the strengths of this study was our use of the TAL interview technique, which allowed us to glean valuable information about how people interpreted what we presented and consequently enabled us to modify the recommendation format to improve understanding. Further, inclusion of a broad range of healthcare professionals who provide direct care to pediatric oncology patients gave voice to users at all levels of the CPG implementation process: from those who select a CPG for implementation to those who deliver direct care. Despite the many strengths of this study, there are limitations to consider. We interviewed healthcare professionals providing care only in the eld of pediatric oncology, and it is unknown whether the results are directly applicable to other elds. As stated previously, pediatric cancer treatment is very protocolized and we might assume that participants would perceive recommendations as non-negotiable. However, we did not nd this and therefore expect that the results of this study could apply directly to any healthcare professional regardless of whether protocols are commonly used. We also did not test whether the nal recommendation format would lead to changes in practice, as we asked participants what they (or their institution) would do if given the recommendation. In future, we could evaluate changes in practice when recommendations using the new format are published. The format was also presented to participants in a package of materials, but not within a website or other electronic platform, such as an electronic order entry system. Although the recommendation format was tested on screen during interviews, further exploration of the format within a platform could provide additional insight to its contribution to CPG-consistent care.

Conclusions
A well-understood recommendation format to present strong and conditional recommendations was created through an iterative interview process with pediatric oncology healthcare professionals. It is likely that this format can be used to convey recommendations to healthcare professionals in other medical specialties. The format is straightforward, making it easy for organizations and CPG developers to use it to communicate their recommendations clearly to intended users. All participants gave permission for their comments to be published in an anonymized form.

List Of Abbreviations
Availability of data and material: No datasets are available from this study owing to the consents given by participants, which limits data to the research team only.