Developing Questions to Assess and Measure Patients’ Perceived Survival Benefit from Adjuvant Endocrine Therapy in Breast Cancer: A Mixed Methods Pilot Study

Purpose This mixed methods study developed multiple question types to understand and measure women’s perceived benefit from adjuvant endocrine therapy. We hypothesis that patients do not understand this benefit and sought to develop the questions needed to test this hypothesis and obtain initial patient estimates. Methods From 8/2022 to 3/2023, qualitative interviews focused on assessing and modifying 9 initial varied question types asking about the overall survival (OS) benefit from adjuvant endocrine therapy. Subsequent focus groups modified and selected the optimal questions. Patients’ self-assessment of their OS benefit was compared to their individualized PREDICT model results. Results Fifty-three patients completed the survey; 42% Hispanic, 30% rural, and 47% with income <$39,999 per year. Patients reported adequate health care literacy (61.5%) and average confidence about treatment and medication decisions 49.4 (95% CI 24.4–59.5). From the original 9 questions, 3 modified questions were ultimately found to capture patients’ perception of this OS benefit, focusing on graphical and prose styles. Patients estimated an OS benefit of 42% compared to 4.4% calculated from the PREDICT model (p < 0.001). Conclusion In this group with considerable representation from ethnic minority, rural and low-income patients, qualitative data showed that more than one modality of question type was needed to clearly capture patients’ understanding of treatment benefit. Women with breast cancer significantly overestimated their 10-year OS benefit from adjuvant endocrine therapy compared to the PREDICT model.


Introduction
Breast cancer is the most common cancer in the United States and modern therapies are very effective with cure rate in excess of 80% [1,2].Adjuvant endocrine therapy is a mainstay of treatment for the 70% of breast cancer patients whose breast cancer expresses Estrogen Receptors (ER) and/or Progesterone Receptors (PR) [3].Adjuvant endocrine therapy has been shown to be effective at reducing breast cancer recurrence and improving overall survival (OS) [4].However, compliance has long been an issue, with only 60% of patients completing the recommended 5 year course of adjuvant endocrine therapy even though higher compliance rates have been shown to improve outcomes [5,6].Medication compliance is also worse among Hispanic patients and those living in rural areas [7,8].Poor compliance leads to increased recurrence and mortality, especially in those with locally advanced disease [9,10].
Conversely, over 90% of women with early stage breast cancer suffer from signi cant side effects related to adjuvant endocrine therapy and a subset of these women derive only a small bene t [11,12].For example, a 65-year-old postmenopausal woman with an estrogen receptor positive, Her2 negative, 1.5 cm breast cancer with no nodal involvement, the 10 year absolute OS bene t from adjuvant endocrine therapy is 0.9% per the PREDICT model [13].That same example with a 5.5 cm tumor and 4 lymph nodes involved shows a bene t of 6.2%.Women with breast cancer may not understand the relatively small bene t of their therapy on survival, as it is well known that patients do not understand many aspects of their medications in general including the actual bene ts [14][15][16].It has been previously shown with other breast cancer treatment decisions that patients trust their physicians to provide the best treatment recommendations, so this may also impact patients' choices to not discontinue their medication despite signi cant side effects [17].In addition, perceptions of risk have also been shown to be related to behaviors including compliance, thus understanding how women view their risk and bene t from adjuvant endocrine therapy is important for decision making [18].Additionally, women from ethnic minority populations and those of lower socioeconomic status often have lower health care literacy in general and thus understanding the bene t from adjuvant endocrine therapy is especially important for these women as they have a higher risk of an inaccurate perception [7,19].
PREDICT is a validated online tool that estimates absolute OS bene t from adjuvant endocrine therapy and is widely used [20].Numerous validated questionnaires exist to assess patient understanding and side effect burden of medication in general but not to assess patient understanding of the perceived bene t from adjuvant endocrine therapy speci cally [21].Data visualization is recognized as an important avenue to assess and disseminate medical information with patient and providers and research for the optimal patient visualization strategy is ongoing [22,23].
We hypothesized that women with early stage breast cancer do not accurately understand the expected OS bene t of their adjuvant endocrine therapy, particularly as estimated by the PREDICT model, and that graphic-based questions would enhance women's ability to accurately estimate their bene t.In preparation to formally test this hypothesis in a larger future study, we performed this pilot study to create and test questions to enhance accurate measurement of women's perceived OS bene t from adjuvant endocrine therapy since these do not currently exist.This pilot also provided preliminary data on participants' perceived survival bene t from adjuvant endocrine therapy.

Methods
A two-phase exploratory sequential study design was employed with patient surveys followed by semistructured cognitive interviews with select participants from the sample to include su cient minority, rural and lower socioeconomic status patients [24].Similar research designs have been commonly used in oncology research to evaluate side effect burden, quality of life (QOL), and patient reported outcomes (PRO) [25][26][27][28][29][30][31].This design was chosen to gather baseline data and to elicit concrete suggestions for the optimal question(s) that would accurately measure participants' perceived survival bene t from adjuvant endocrine therapy which may not have been captured with quantitative surveys alone.Surveys.Surveys were developed to collect self-reported demographics including race, ethnicity, urban vs rural status, income, education level, age, adjuvant endocrine therapy medication, and the last time the patient discussed adjuvant endocrine therapy with their provider.Additional survey components included the PROMIS Self-E cacy for Managing Medications and Treatments questionnaire short form [32,33] to evaluate patient con dence in choosing, managing and gathering information about medications along with a brief validated questionnaires to assess health literacy by Chew et al [34,35].Surveys were distributed by research coordinators to patients at the time of a regularly scheduled oncology follow-up visit and included informed consent, a study explanation, and the opportunity to decline.The surveys were completed via preloaded tablets using the REDCap platform or paper forms depending on patient preference.Spanish language versions of the survey were available in both formats and were translated by a certi ed Spanish translator.Data extraction.Data extraction done by research coordinators from the Electronic Medical Records (EMR) included age, past medical history, tumor characteristics, type of adjuvant endocrine therapy, discontinuation reason (if applicable), menopausal status, and PREDICT model results based on that individual patient's clinical situation.

This study enrolled
Interviews.At the conclusion of the survey, patients could volunteer to be contacted for a phone or inperson interview.Interviewees were chosen using purposive sampling to represent a mix of rural/urban, Hispanic/Non-Hispanic, and different income levels.The interviews were completed by a Spanish and English bilingual member of the research team who had experience working with the diverse New Mexican population.The goal of the interviews was to re ne and nalize survey questions designed to assess patient understanding of the bene ts of adjuvant endocrine therapy.The nine questions we tested were drawn from multiple styles based on the PREDICT outputs and the COMET study [36] and included prose, ll in the blank, numerical, non-numerical, percentages, iconographs, and bar graphs (Fig. 1a).The research team developed semi-structured interview guides based on domains of interest, such as clarity and understandability, adapted from the literature [37].The cognitive piloting occurred in three iterative rounds (Fig. 2) followed by consensus discussion meetings amongst the study team to determine next steps.
All interviews were conducted via Zoom with the camera on or in person.This allowed the interviewer to identify and respond to visual cues in body language.The interviewer asked two questions at the start of each interview, to gauge general understanding of the participant's knowledge and priorities on the difference between cancer recurrence and dying from cancer.The interviewer then showed the test questions one by one.After reading each question to the participant, the interviewer rst asked the participant to select the best answer to the question and talk through their thought process aloud.The interviewer then asked the participants to rate the question on two Likert scales from 1-7 on clarity and understanding.For example: "How clear was this question for you?" (1 = not clear at all, 7 = totally clear).Opinions were gathered through probing questions on preference of question and ways to improve test questions for clarity and understanding.The interviewer was experienced in conducting cognitive interviews and did follow-up any verbal or non-verbal reactions including hesitations, spontaneous comments, or expressions such as confusion or uncertainty per published literature [37].Interviews lasted an average of 45 minutes.Participants received a $25 gift card for their time.We used feedback from all 27 interviews to select and make edits to the questions.
Focus Groups: We then conducted two focus groups (FG1 and FG2) to assess which question(s) would be the clearest and elicit the most accurate answers.We recruited participants who had already completed a cognitive piloting interview to participate in the focus groups.Focus groups were conducted only in English and via Zoom; they lasted 90 minutes each and participants were compensated with a $25 gift card for their time.Three test questions selected and modi ed from the interviews were the focus of each FG.Two facilitators moderated each discussion and it was explained to the participants that the original nine test questions had been modi ed or eliminated and why.We explained that future changes to the nal three questions would be based off participants' responses in both FGs.Approximately 30 minutes were spent discussing each test question.The nal few minutes were spent gathering answers to the same overarching questions as in the interviews: "Would anything change if we asked about recurrence instead of dying and what if we asked about increasing chance of survival instead?"Changes suggested by FG1 were discussed during FG2.

Consensus
As demonstrated in Fig. 2, consensus was an iterative process.Following each round of participant interaction, the study team met to discuss results and participant comments.The initial consensus process allowed us to eliminate six questions.The remaining three questions were re ned during the secondary consensus process.During the nal consensus following all data collection, the study team made nal phrasing and layout changes to the three questions (Fig. 1b).The decision was made to include three questions instead of just one to allow for reproducibility of answers and because many patients seemed to prefer one question type over another but no one question seemed best suited for all participants.
Endpoints.The primary endpoint of this pilot study was to create questions to accurately and clearly measure women's perceived OS bene t from adjuvant endocrine therapy to be used in a future study.The secondary endpoint is to measure whether or not women who initiated, declined, were not recommended, or discontinued the therapy are accurate in predicting their OS bene t compared to PREDICT modeling.Other secondary endpoints were to examine health care literacy, understanding of ability to manage medications and relationship of time interval between last discussions of adjuvant endocrine therapy bene t with accuracy of patients' predictions.Data analysis.The results of the cognitive piloting activities were primarily descriptive in nature.The input provided about each of the various question types from the individuals who engaged in interviews was summarized in tabular form, and counts and percentages were calculated.The percent of participants who provided a positive endorsement of each candidate question type was estimated.We compared a ranking of these positive endorsement proportions across question type.The data from the study surveys and the data obtained from the chart review, were descriptively summarized (means, standard deviations, etc., for quantitatively scaled data, and counts and percentages for categorical data).This included categorizations of the patients' responses about the likely bene t from adjuvant endocrine therapy.Participants' ratings of each question's clarity and understanding were compared, after transformation, using linear mixed effects regression to test for differences in ratings across questions, and to identify those questions with the highest clarity and understanding, while accounting for withinperson correlations.Additionally, we combined the ratings of which questions were selected to be the most or least clear into a single perceived clarity score that ranged from positive 100% (equivalent to all respondents selecting the question as being most clear) to negative 100% (equivalent to all respondents selecting the question as being the least clear), and compared these scores across the nine questions using linear mixed effects models.The difference between the patient-assessed likely bene t of treatment with adjuvant endocrine therapy was compared to the PREDICT-estimated likely bene t was assessed using a paired t-test.A further assessment of the degree to which breast cancer patients correctly identify the likely bene t of treatment with adjuvant endocrine therapy was obtained by estimating Lin's concordance correlation coe cient [38].This correlation coe cient is equal to a value of 1.0 when two ratings result in equal values, and is equal to zero when there is no concordance between the two ratings.

Results
Surveys.A total of 53 patients completed the initial survey of which 42% were Hispanic, 30% rural, and 47% received $39,999 or less in household income per year (Table 1).Patients answered 88.6% of the questions asked.Patients reported average con dence about treatment and medication decisions with 70% very con dent they can actively participate in treatment decisions, 58% very con dent they can use their own judgement regarding treatment alternatives or not having treatment with mean score of 49.4 (95% CI 24.4-59.5)compared to a mean of 50 (standard deviation of 10) for United States general population (Fig. 3).Additionally, 61.5% of patients reported adequate health care literacy (Supplemental Table 1).
Interviews.Of the 53 patients who completed the initial surveys, 40 (75.5%)indicated willingness to participate in the interview, and 27 (67.5%)interviews were completed.The 13 potential participants who were not interviewed were not reachable or had scheduling con icts.Demographics were similar for those interviewed with 44% Hispanic, 33% rural, and 47% with $39,999 or less in household income per year (Table 1).The 27 patients ranked all questions from 1-7 in terms of clarity and understanding.Linear mixed effects regression suggested that the respondents rated the nine questions differently according to their clarity and understandability (p < 0.001).Question #3 was a prose-based question and was the highest rated question with 6.8 (SD = 0.5) for clarity and 6.6 (SD = 0.8) for understanding.The next highest ranked question was #6, which used pictograms and was rated 6.6 (SD = 0.6) for clarity, and 6.6 (SD = 0.5) for understanding.Question #6 (n = 14) and Question #7 (n = 12) were the most frequently chosen as most clear.Question #7 had a signi cantly higher clarity score of 37% than those of questions #3, #8, #1, and #2, which had clarity scores of 0%, -11.1%, -14.8%, and − 18.5% respectively.The two questions most frequently chosen as least clear were Question #1 (n = 9) and Question #8 (n = 8).We found that there were signi cant differences in perceived clarity among the nine questions (p = 0.02).Each of these adjusted mean scores shared the same pooled standard error estimate of 12%.While assessing the questions during these interviews, three larger categories of problems arose: problems with the question wording, problems with the answer choices, and problems with the visual (Table 2).
Each of the questions assessed in this pilot study attempted to prompt participants to estimate the perceived bene t that adjuvant endocrine therapy might provide to them.We compared the resulting selfratings to the values calculated using the PREDICT model.Across the various question types, the patients estimated a signi cantly higher average bene t from adjuvant endocrine therapy on survival, at an average level of 42% (SD = 14.6) when compared to the PREDICT model which showed an average bene t of 4.4% (SD = 4.3) (Fig. 4, p < 0.001).Across the question types, the mean patient answer was consistent except for question 1, which was non-numerical prose.For questions 2-9, patients consistently estimated higher survival bene t than the PREDICT model (p < 0.001 for all).The most common answers for Question 1 was that adjuvant endocrine therapy prevent people from dying "A modest amount" (22%) and "A lot" (22%).There was no relationship detected between the question answers and when the patient last discussed adjuvant endocrine therapy with their oncologist when comparing 0-6 months vs more than 6 months ago (p = 0.22).Comparing the survey averages of the expected treatment bene t to those calculated using the PREDICT tool resulted in an estimate of Lin's concordance correlation coe cient of -0.003, indicating essentially no agreement between the breast cancer patients' ratings and those calculated using PREDICT.
All participants understood the difference between the cancer recurring and dying from cancer.Many participants stated that they thought more about recurrence, because they would not die from cancer unless it recurred rst.We also asked participants if they would interpret the questions differently if we asked about recurrence instead of dying.This question was also often followed up by, "what if we asked about increasing survival instead?"Some participants preferred being asked in terms of dying because it was more straight-forward and "didn't beat around the bush".Other participants preferred the survival wording because "you see the word death a lot" and because "it gives you hope".
Patient suggestions for how to improve patients' understanding of the bene ts of adjuvant endocrine therapy included giving participants a brochure or visual aid when they leave, teaching participants more about general medical knowledge, not being too technical, and asking medical team members to be selfaware of the difference between their own risk tolerance and their patient's risk tolerance when recommending treatment.

Discussion
The mixed methods study design, featuring patient surveys followed by in-depth qualitative interviews, provided nuanced understandings and preferences of these patients in underrepresented ethnic minority groups, rural and low-income patient populations.The interviews revealed that different patients perceived different questions types as being more or less clear.While most patients preferred a visual representation such as the iconograph (question #6) or bar graph (question #7), a signi cant minority gave that question type negative feedback and instead preferred a prose format that made the question personal by directly asking about their risk reduction (question #3).This suggests that our hypothesis that visual representation would be superior is partially correct and multiple question formats may be needed to ask patients important research questions to ensure clarity for all patients as shown in the three questions that were the nal product of this study.Taken a step further, patient facing materials may ideally be represented in multiple formats to increase understanding of the risks and bene ts of therapy.This has implications in both the research and clinical settings.
Patients signi cantly overestimated their 10-year OS bene t from adjuvant endocrine therapy compared to the estimates from the PREDICT model (42% vs 4.4%, p < 0.001).Furthermore, the patient-estimated levels of average bene t demonstrated essentially no agreement with the estimates from the PREDICT model.These ndings are consistent with our hypothesis that patients do not accurately understand the OS bene t from adjuvant endocrine therapy.These patients reported adequate health care literacy along with average con dence in their medication management and treatment decisions.This adequate con dence regarding their health care literacy and average con dence in medication management is contradicted by the inaccuracy of the patients' estimates of their OS bene t.This phenomenon of patients with con dence in their inaccurate predictions is not well described in the literature and is important for clinical care and a topic for further research.A more general trend of people being overcon dent in their decisions is a well-known phenomenon which is likely related [39,40].
Patients often do not fully understand the associated bene ts and side effects of their medications [41,42].They often do not fully grasp the risks and bene ts of medical interventions which has been well documented in research surrounding consent for surgery and clinical research [43].Improving questionnaires meant to assess patients symptoms and ask general medical questions are an active area of research that is showing promise with the addition of graphic visualization [44,45].Our study adds to this body of evidence showing that women with hormone receptor positive breast cancer signi cantly overestimate the OS bene t from adjuvant endocrine therapy.This research also echoes previous ndings that visual representation of the bene ts of medications are easier to understand for most patients and adds to the literature that some patients nd prose more understandable.A combination of these two question types may be the best approach.This is an important area for future research as patients who fully understand the bene ts and risks of their medications can make truly informed decisions.Additionally, patients who are con dent taking their medications are more likely to be adherent [46,47].Since poor adherence is a signi cant problem with adjuvant endocrine therapy leading to worse outcomes, improvement in adherence could markedly improve patient outcomes.Conversely, adjuvant endocrine therapy has signi cant side effects that can affect quality of life [48,49].Patients who have a low bene t may suffer these side effects without considering discontinuation because they believe adjuvant endocrine therapy to be much more bene cial than it actually is for them.Improving how we assess patient understanding could lead to targeted patient education which could improve outcomes for patients and potentially encourage patients to be more engaged in their health care decisions.
This study focused on OS primarily because it is considered the gold standard outcome measure in oncology.However, both OS and recurrence are important to patients as illustrated in the interviews.Similarly, our study does not take into account the signi cant bene t from adjuvant endocrine therapy as chemoprevention related to a subsequent cancer in either breast.Patients were aware of the difference between recurrence and OS, suggesting that this can be discussed with patients in the clinic when assessing the bene ts of treatment.This is important clinically as many treatments reduce recurrence risk but have not demonstrated improvement in OS and these nuanced risk/bene t conversations can be daunting for patients and providers.The qualitative data from this study would encourage clinicians to have these nuanced conversations as patients are likely to comprehend the difference between recurrence and OS.
This study has inherent limitations.Certain populations such as African American and Asian patients were not well represented in this study, which reduces the generalizability of our ndings.This work represents a single academic Comprehensive Cancer Center with a small number of providers and may not be broadly applicable to other sites or practice types.While a variety of question types for assessing patient understanding of adjuvant endocrine therapy were selected for this study based on the PREDICT model and the COMET study, potentially superior question types may have been missed for inclusion.Patients may have been thinking of and providers may have discussed relative bene t and not absolute bene t which may have led to inaccurate measurements as we measured absolute bene t in this study which we tried to make clear in the questions asked.Problems with numeracy may have also led to inaccurate results but graphical representation of questions and having multiple questions asking the same concept hopefully diminished this effect.Some women who are experiencing side effects may also report a higher bene t because they are in fact reporting the bene t they would need to have to continue to endure the side effects of their adjuvant endocrine therapy as this was documented in two interviews.
The patient survey data were self-reported and subjective, and therefore may re ect self-reporting biases and/or inaccurate information.The surveys used were a combination of validated surveys and questions created solely for this study which were not previously piloted, so there may be limitations in reproducibility.In addition, the sample size was relatively small so small differences may not have been detected.
In this group of breast cancer patients with substantial representation of ethnic minority, rural and lowincome groups, qualitative data showed that more than one modality of question type was needed to clearly capture participants' perceived survival bene t from adjuvant endocrine therapy including both visual and prose-based questions.Patients also signi cantly overestimated their 10-year OS bene t from adjuvant endocrine therapy compared to the PREDICT model.Using this study's results, these selected and modi ed questions will be used in a larger study of this patient population to con rm the above ndings, examine differences among subgroups, assess providers' estimate of bene t and providers' estimate of patient accuracy.Future research should examine educational interventions to improve patient understanding of their adjuvant endocrine therapy, as improved understanding may support a more accurate risk / bene t assessment impacting adherence, informed treatment discontinuation and de-escalation strategies.• Question 5: "increase that the chance the woman is alive" it doesn't seem like a phrase I would hear, it's less selfexplanatory" • Question 5: "The last part of the question that I had to read over again -the question itself was a little clunky and I had to read it twice." We eliminated the last part of the question, and kept the context from the rst of the question.
• Numerous Questions: Many participants found the questions more clear when we added 'you' or 'your' to the question.
Questions were updated with you or your when appropriate.
• Numerous Questions: "Maybe when you say hormone treatment, instead say hormone pill treatment -instead of saying therapy you should say hormone pills." In the nal questions we used the terms "hormone pill treatments" and "hormonal therapy pills." Answer Choices • Question 3 and 5: "I don't like them because of the percentage -people are not going to grasp it -I don't like the answers." • Question 4: "The question is hard to understand.What does 1 out of 100 mean or does it mean all the people through 100?I don't understand." Added both percentages and n out of 100 to all answer choices.
• Question 1: "#1 is the least clear, what I call a very little could be different for someone else." • Question 1: "#1 was the worst because it didn't quantify anything." Eliminated question #1.
• Question 2: many participants hesitated to answer, or responded with a number out of 20 followed by a percentage that didn't match.For example, one participant said they thought it was 2 out of 20, and then later said that would be 5%.

Graphic Elements
• Question 8: "I don't think I understand the continuous graph in the context of this question.You are either bene tting or not.I don't like this visual presentation of the answer options." Eliminated Question #8.
• Question 7 and 8: Many participants did not like the color of the orange and blue.Some participants noted that the orange really highlighted how many women won't bene t from treatment.
Changed the colors to pink to represent patients who bene t, and grey shading for the patients who don't.Results of PROMIS English and Spanish speaking patients at the University of New Mexico Comprehensive Cancer Center from August of 2022 to March of 2023.The University of New Mexico (UNM) Health Sciences Center's Human Research Protections O ce [UNM HRPO] (#19-562) approved the study.The study enrolled women with a history of stage I-III hormone receptor positive breast cancer who were eligible for any adjuvant endocrine therapy; Tamoxifen, Anastrazole, Letrozole or Exemestane.The women must have initiated, declined, were not recommended, or discontinued the therapy within the last 5 years and be 18 years or older.Patients with Ductal Carcinoma in Situ, men, and women on adjuvant endocrine therapy beyond 5 years were excluded.

Figure
Figure

Figure 2 Patient
Figure 2

Table Problems with
Questions and Changes Made to Questions Based on Interviews.