Remote Mentoring Optimizes Virtual Collection of Patient-Reported Data: A Prospective Cohort Study with Adaptive Design Conducted in COVID-19 Era.

Prior to the COVID-19 pandemic, patients attending ambulatory clinics at cancer centers in Ontario completed the Edmonton Symptom Assessment Scale (ESAS) at each visit. At our center, completion was via touch pad, with assistance by clinic volunteers. As of March 2020, clinic appointments were conducted virtually when possible and touch pads removed. We anticipated a negative impact on the collection of patient-reported outcomes (PROs), and the recognition of severe symptoms. Methods: We performed a prospective cross-sectional cohort study to test the feasibility of remote ESAS completion by patients with appointments at a weekly surgical oncology clinic. Patients in the initial study cohort were asked to complete and return the ESAS virtually(V). Given low completion rates, the ensuing cohort was asked to complete a hard-copy(HC) ESAS. For the nal cohort, we used an adaptive approach, providing remote, personal mentorship by a member of the care team to support virtual electronic ESAS completion (virtual-mentored(VM) cohort). Results: Between May-July 2020, a total of 174 patient encounters were included in the study. For the V cohort, 20/46 patients (44%) successfully completed and returned the electronic ESAS, compared to 49/50 (98%) for the HC cohort. For the VM cohort (n=78), the completion rate was 74%. Questionnaire completion was not predicted by age, sex or tumor site, although patients who completed the ESAS were more likely to be in active management rather than surveillance(p=0.04). Of all completed forms, 42% revealed a depression score ³2, and 27% an anxiety score ³4. Conclusions: We identied signicant barriers to the virtual completion of ESAS forms, with a lack of predictive variables. The severe degree of psychological distress reported by ~50% of respondents demonstrates the need for ongoing regular collection/review of these data. Innovative solutions are required to overcome barriers to virtual collection of PROs.


Introduction
Over the past decade, the importance of collecting Patient Reported Outcomes (PROs) alongside objective data regarding cancer status has gained widespread recognition amongst clinical oncologists and interdisciplinary cancer teams [1,2]. PROs provide a measure of the value of investigations and therapies to the individual patient, assisting with decision-making in the acute phase of cancer management [3][4][5]. PROs also indicate the burden of symptoms, both physical and psychosocial, felt by the cancer survivor[6-8]. The Edmonton Symptom Assessment Scale (ESAS) is a validated self-reporting tool of symptom severity for nine common symptoms of advanced cancer that accurately captures PROs including a variety of physical symptoms (pain, tiredness, nausea, drowsiness, appetite, wellbeing, shortness of breath) and psychological distress (depression, anxiety), with the option of adding a tenth patient-speci c symptom [9][10][11]. There is good evidence that PROs captured through ESAS completion more accurately depict symptom burden than do interrogation and documentation by a clinician [12,13]. In view of this, completion of the ESAS by all patients attending cancer clinics has been mandated in many jurisdictions, including the Province of Ontario, Canada [14,15].
Prior to March 15, 2020, ambulatory patients seen at Princess Margaret Hospital, Toronto, completed the ESAS at each clinic visit. Compliance exceeded 85% at our institution in the immediate pre-COVID era. Patients used touch pads provided by the clinic to enter their scores for each symptom category, using an 11-point scale from 0 (no symptom) to 10 (worst possible symptom) [16]. Patients who had any di culty navigating the platform, for any reason, were assisted in person by clinic volunteers. This direct electronic patient entry then prompted real-time review by clinicians. Scores that indicated clinically signi cant symptoms (³2 for depression; ³4 for all other symptoms) triggered follow-up by the primary oncology team and referral as per Cancer Care Ontario practice guidelines to the appropriate health care providers, for example, a psychiatrist specializing in care of oncology patients [17]. Similar systems have been implemented across cancer clinics in Canada and the United States [18][19][20][21].
The advent of the COVID-19 pandemic dramatically altered the care of cancer patients world-wide. Early reports implicating visits to the cancer center in transmission of the SARS-Cov2 virus to vulnerable cancer patients [22,23] triggered severe measures to curtail in-person clinic visits. This resulted in widespread adoption of virtual assessments, both for initial consultations and for follow-up visits. For patients who did attend ambulatory clinic in-person, many restrictions were immediately implemented. In particular, at our institution touch pads were removed from clinics. Collection of PROs via ESAS completion, so recently championed and supported, was abandoned. While this precaution was at the time understandable given the concern for patient safety, we wondered what impact these physical safety measures might have on our ability to detect signi cant psychological distress in cancer patients who were already facing delays in investigation and treatment, as well as alterations in routine surveillance protocols. In an attempt to ll this gap, we introduced virtual ESAS completion, but in an initial cohort of patients found that implementation was not successful. After demonstrating near-universal completion of hard-copy ESAS by patients visiting clinic in person, and identifying barriers to receipt and completion of virtual ESAS forms, we developed a personal virtual mentoring program and showed that this signi cantly improved completion rates. We also here report experience of severe levels of psychosocial distress in ~50% of ambulatory cancer clinic patients amidst the COVID-19 pandemic.

Methods
The primary objective was to evaluate the rate of ESAS form completion through virtual means. Secondary objectives were to determine if associations existed between patient characteristics and virtual ESAS questionnaire completion, and to describe barriers to virtual collection of PROs. This study was approved by the Princess Margaret and Mount Sinai Hospital REBs.
ESAS questionnaire. The ESAS [9] is a validated brief, practical, and comprehensive self-reporting tool of symptom severity for nine common symptoms of advanced cancer (pain, tiredness, nausea, depression, anxiety, drowsiness, appetite, wellbeing, shortness of breath), with the option of adding a tenth patient-speci c symptom; higher scores represent worse symptom intensity [24]. A llable PDF version using the Cancer Care Ontario (CCO) form (cancercareontario.ca), with clickable buttons associated with the numerical scale values for each symptom was generated for virtual dissemination. Only page 1 of the form was used for the present study. Overall assessment of symptom severity was generated by adding individual symptom scores to generate the ESAS total symptom burden score. The total symptom burden score and individual symptom scores were classi ed as absent (0), mild (1-3), moderate (4-6), and severe (7-10) using de ned cutoffs from the literature, with clinically signi cant scores de ned as scores of > 30 (total symptom score), ³ 4 (individual symptom scores) and ³ 2 (depression symptom score) [25][26][27][28].
Study Population. Consecutive patients attending a virtual or in-person appointment at a mixed cancer site surgical oncology clinic were approached for inclusion in this study, beginning in May 2020. Our cancer center had by then undergone a major transition to virtual clinics due to the COVID-19 pandemic. Patients had to be ³18 years and able to provide informed consent. Patients were enrolled into three sequential cohorts, according to the platform used for ESAS completion.
For the rst cohort (Virtual), patients were contacted by phone shortly after their appointment, whether it was virtual or in-person. Their permission to email the llable PDF ESAS form was requested, and they were asked to return the completed form electronically. Patients who could not be reached with an initial phone call were called up to 2 more times. The ESAS form was emailed to the address provided by the patient. If the form was not completed and returned within one week, an email reminder was sent.
For the second cohort (Hard-copy), patients who attended in-person appointments were asked to complete a hard-copy ESAS form in clinic. The completed form was collected at the time of the appointment.
For the nal cohort (Virtual-mentored), an adaptive strategy was employed whereby a research team member provided remote support to facilitate ESAS completion. Patients were approached at the time of virtual or inperson clinic visits, and those who agreed to participate provided an e-mail address and phone number at which they could be contacted. On the same day, they were sent an e-mail with a clickable PDF attachment of the ESAS survey and a note that explained that they would receive a follow-up call to discuss their experience with the survey. Patients were called within 3 days of sending the survey. If they had successfully completed and returned it, the caller explored their experience in completing the survey, and took the opportunity to address any elevated scores on the ESAS form. Any unaddressed concerns were brought to the attention of the most responsible treating physician. If they had not completed the survey, the caller reminded them to complete it, and veri ed the contact email address. Up to 2 additional reminder calls were made.
All patients in the Virtual and Virtual-mentored cohorts were asked about barriers to electronic completion and return of the clickable PDF. Patients who stated that they did not receive it were asked to check spam or junk mailboxes, and the survey was resent. Messages were not left on an answering machine/voice mail or with a third party.
A total of 6 patients (4 and 2 in the Virtual and Virtual-mentored cohorts, respectively) required their responses to be transcribed by a research team member over the phone, as they were unable to understand how to complete the PDF and submit it; these individuals were analyzed within their assigned cohort, on an intentionto-treat basis. In the Virtual-mentored cohort, 10 out of 58 patients who completed the ESAS questionnaire did so by means of a hard-copy form that they obtained in clinic at their request; these patients were analyzed within the Virtual-mentored cohort.
Statistics. Summary statistics were used to describe the demographic and clinical characteristics of the study population. Statistical signi cance was assessed by Student's t tests for continuous variables and Chi-squared (Fisher's exact) test for categorical variables using Prism software (GraphPad Software, La Jolla, CA). A p value <0.05 was considered signi cant. Error bars re ect median and IQR.

Results
Description of study participants. A total of 174 patient visits to a complex surgical oncology ambulatory clinic between May 27 and July 29, 2020 were included in the study ( Table 1). The median patient age was 62 (range 19-90) and 53% were female. The primary tumor site was gastrointestinal in 31%, soft tissue in 48%, and "other" in 21%. Recruitment into the three cohorts (Virtual, Hard-copy, Virtual-mentored) was sequential based on clinic date, with cohort size adapted to the observed ESAS completion rate as the study proceeded (see Methods). Age, sex and tumor site did not differ between the three cohorts ( Table 1). We further characterized patients according to the phase of their cancer journey at the time of the study: 41% of patient visits were within a phase of Active investigation or treatment, 50% were in Surveillance following treatment, and 9% were while receiving symptom-directed care for cancer that was not considered curable (Palliative). There was a signi cant difference in the distribution of management phase between the cohorts. Patients in the Hard-copy cohort were more likely to be undergoing Active investigation/treatment (54%) or receiving Palliative measures (20%) and less likely to be in Surveillance (26%) than those in either the Virtual or Virtual-mentored cohorts (p<0.05, c 2 ).
ESAS completion rates. Of the 174 ambulatory visits, 127 culminated in ESAS completion by the patient; three patients formally declined to complete the ESAS form ( Fig. 1). At the start of the study, a research team member contacted the patient by phone shortly after their virtual or in-person visit with the physician. The ESAS form was then emailed to an address provided by the patient. For the rst cohort (Virtual), 46 patients were approached over a three-week interval, 15 of whom did not respond after up to 3 phone calls and 3 emails. One patient declined to complete the form (Supplemental Table 1). The ESAS form was emailed to the remaining 30 patients, together with instructions to complete and return the form electronically. While 20 patients did complete the survey, 4 of these requested to complete it via phone transcription with a member of the research team, as they stated that they had di culty handling the PDF. The overall completion rate for this Virtual cohort, analyzed by "intention to treat", was 44%, and ~60% of the non-completions re ected upfront lack of response from the patient. Postulating that the low completion rate might be related to the virtual format and remote relationship between study participant and the research team, we reverted to the hard-copy format initially employed when ESAS forms were rst rolled out at our institution in 2007. Over the next 3 weeks, 50 patients who visited the clinic in person were asked to complete a printed ESAS form (Hard-copy cohort). 49 (98%) agreed and did so; one patient agreed to complete the form, but deferred to the end of the appointment and did not return it.
Given the notable discrepancy in completion rate between the two initial cohorts, we speculated that signi cant barriers to patient handling of the virtual ESAS form might have existed. Using an adaptive approach, we created an intervention cohort, offering patients "virtual mentoring" to assist with the practical aspects of questionnaire completion and return. For this nal Virtual-mentored cohort, 78 patients who attended clinic inperson or virtually were approached over a 4-week interval. 2 formally declined to complete the form, and 18 of the 76 who had originally agreed to complete the form did not ultimately return a completed ESAS, 2 returning a blank form, and 16 not returning a form despite verbal offer of mentorship. Of the 58 patients (74%) who successfully completed the form, two requested to do so via phone transcription, dictating their responses to the research team member, and ten requested to complete a hard-copy form in clinic (Fig.1).
We queried patient-and disease-related variables that might be associated with ESAS completion. Analysis of all patients in the three cohorts grouped together showed that completion was not predicted by age, sex or primary tumor site (Table 2). Patients who did not complete were less likely to be under active investigation/treatment than those who did (28% vs. 46%, p=0.04, c 2 ). Analysis of the subgroup of patients included in the Virtual and Virtual-mentored cohorts (n=124) also showed that questionnaire completion was not predicted by age, sex, or primary tumor site, nor was it predicted by management phase (Table 3).
ESAS patient reported data. Of the 127 ESAS forms completed by patients in all three cohorts, 117 (92%) reported at least one symptom score ³1, indicating the presence of a symptom. The overall responses for each symptom scale are shown in Fig. 2A, categorized by symptom severity. The symptom with the highest prevalence of clinically signi cant severity was depression (42% had a score ³2), followed by wellbeing (35% ³4), tiredness (31% ³4) and anxiety (27% ³4). The ESAS total symptom burden score (>30: moderate-tosevere vs. £30: absent-to-mild), as another measure of clinically signi cant distress, showed that 20% of respondents (26/127) had a moderate-to-severe total symptom burden (Suppl Table 2).
There was no difference in the ESAS total symptom burden score reported between the three cohorts (>30 in 20%, 22%, 38% of Virtual, Hard-copy and Virtual-mentored cohort respondents, respectively). Similarly, there were no signi cant differences in individual symptom scores reported for each cohort, as illustrated by the distribution of scores for depression, wellbeing, tiredness and anxiety (Fig. 2B-E) and the remaining individual symptoms (Supplemental Figure). High total symptom burden was not predicted by age, sex or management phase, although patients with high total symptom burden were more likely to have a primary GI tumor (50% vs. 28% of those with low symptom burden, p=0.02, c 2 ) (Suppl Table 2).
In nine patients (2 Virtual; 1 Hard-copy; 6 Virtual-mentored), review of symptom severity scores by a member of the research team prompted follow-up conversations to validate and clarify the reported symptoms. Six patients were having signi cant, poorly controlled physical symptoms, and with patient consent this was directly communicated to the most responsible physician by a member of the research team. Three patients who had high anxiety and depression scores were questioned further: one had an established relationship with a psychiatrist, and two were offered referral to psychosocial support services at the cancer center. In particular, review of the ESAS forms triggered an intervention in 4 of 6 Virtual-mentored patients with high scores.
Barriers to virtual completion of patient reported outcome questionnaire. Of 106 visits that yielded an agreement by the patient to virtually complete and return an ESAS form, 28 did not culminate in completion (10/30 and 18/76 in the Virtual and Virtual-mentored cohorts, respectively). Some patients in the Virtual cohort stated that they were unable to open, complete or save the PDF, and others reported generalized "technologyphobia". Of the subgroup of patients in the Virtual-mentored cohort who completed the form, the majority felt it was a straightforward process, and had no di culty. The few individuals who had challenges with handling the PDF were able to overcome them with the help of a research team mentor.

Discussion
This study demonstrates the existence of signi cant barriers to virtual collection of PROs in 25 to 50% of patients attending a complex surgical oncology clinic. Explicitly stated and inferred barriers were partially addressed by provision of personalized remote mentoring to facilitate successful electronic handling of the llable form by patients. Importantly, completed ESAS forms revealed a high rate of severe psychosocial distress amongst ambulatory patients being treated/followed through the cancer clinic amidst the rst wave of a global pandemic. Anxiety regarding treatment delays and depression related to social isolation have been cited as COVID-related sources of psychological distress that persist even now [29][30][31].
A body of research rst disseminated in the palliative care literature clearly demonstrates that during a clinical encounter in the ambulatory clinic, treating clinicians commonly overlook symptoms that are impinging signi cantly on a patient's quality-of-life (QoL) [6, 32,33]. Prospective studies have shown that patient satisfaction and QoL are objectively improved by routine PRO capture in cancer centers [5,34]. In the early 2000s, the evidence-based movement to measure, review, and target PROs gained su cient strength that this practice had become standard in major cancer centers prior to the COVID 19 pandemic.
Within the Province of Ontario, Canada, routine administration of the Edmonton Symptom Assessment Score (ESAS) questionnaire was implemented in ambulatory clinics at all regional cancer centers in 2007 [11]. Implementation expanded to include many of the partner community hospital sites, and ESAS data were integrated into many of the local EMRs. In 2015, the Ontario Cancer Plan IV speci cally stressed the importance of PROs. Real-time electronic capture of PROs at each clinic visit was effected by directing the patients to a computer work station or tablet following registration. The output was then given to the clinical team to use within or after the clinical encounter with the physician. Both provider-facing and patient-facing symptom management guides for each symptom found in ESAS were used as guidance for the clinical care team as to how to respond to elevated symptom scores (www.cancercareontario.ca/en/symptommanagement).
Even in the pre-pandemic era, various barriers to ESAS/PRO completion were described. These include insu cient time, challenges with language uency/literacy, misinterpretation of ESAS terms, di culties in rating current symptom level, and technology issues including access/pro ciency [35][36][37]. As in many cancer centers pre-pandemic, we provided patients with on-site personal support from clinic staff and/or volunteers, to enhance PRO completion.
Sweeping changes were quickly implemented within North American ambulatory cancer clinics as the rst reports of high rates of Sars-Cov2 transmission and serious illness in cancer patients emerged from centers in China and Italy [38][39][40]. At our center, as at many others, routine ESAS completion was abandoned for over 18 months. The high rates of severe depression and anxiety revealed by ESAS administration, whether virtual or hard-copy, con rm the value of this tool and highlight the importance of continuing to collect PROs during turbulent times. Furthermore, though we had conjectured that high symptom burdens would be seen largely in the perioperative period and in patients undergoing other active investigation/treatment, our results indicate that neither age, gender, diagnosis or treatment stage were predictive of symptom burden. Again, the value of PROs was demonstrated and must be championed as a priority.
While the remote mentoring program we implemented here did result in improved ESAS completion rates, barriers remained. Within the framework of this study, it was not possible to ascertain what barriers, if any, were encountered by non-responding patients. Health literacy and/or language pro ciency may have played a role [35][36][37]41], but these variables were not captured in our study. We acknowledge that we may have missed PROs on the most vulnerable of our cancer patients amidst the COVID pandemic [42][43][44]. Other barriers to virtual care cited by patients and providers have included poor reliability of caregiver assessments, provider burden, and lack of understanding regarding the timeframe of assessments [35,37].
Virtual mentored ESAS form completion allowed patients to have an opportunity to report on symptoms after their clinic visit. For many patients, this was a brief interaction and the form was easy to complete. A unique component of this format was that it allowed for a second interaction after the clinic visit and therefore some unaddressed or persistent symptoms or concerns could be brought to the attention of the care team, allowing for immediate action to be taken. Organization and conduct of the virtual mentoring interaction could be time intensive. All patients required at a minimum one phone call, but some had several reminder phone calls.
Although most calls were only 1-2 minutes in length, for those with questions or concerns, these calls took up to 10 minutes. In this study, a physician was completing the calls and was able to answer questions, however this is a resource that is not always readily available.
The high completion rate observed with in-person Hard-copy ESAS questionnaires suggests that time and accessibility were factors; time spent waiting for the physician in clinic was readily directed towards form completion. We note the signi cant minority of patients who did not wish to participate in virtual form completion, regardless of the availability of mentoring, but were willing to be guided through the form and convey their responses verbally to a team member. This indicated a degree of "technology-phobia" that did not  Figure 1 Diagram depicting study work ow and patient response over the study period. Distribution of ESAS symptom scores in study patients. a) Proportion of patients with ESAS scores in the following severity categories: unknown, no symptoms (0), mild-to-moderate symptoms (1-3), moderate-tosevere symptoms (4-6), and severe symptoms (7-10), by symptom type, in the total cohort of 127 patient encounters that resulted in ESAS questionnaire completion. Depression score categories (right) were as follows: unknown, no symptoms (0), mild symptoms (1), severe symptoms (2-10) [27]. b-e) Distribution of patient scores for depression (b), well-being (c), tiredness (d), and anxiety (e), displayed according to assigned