Patient and clinician perspectives of a remote monitoring program for COVID-19 and lessons for future programs

COVID Watch is a remote patient monitoring program implemented during the pandemic to support home dwelling patients with COVID-19. The program conferred a large survival advantage. We conducted semi-structured interviews of 85 patients and clinicians using COVID Watch to understand how to design such programs even better. Patients and clinicians found COVID Watch to be comforting and beneficial, but both groups desired more clarity about the purpose and timing of enrollment and alternatives to text-messages to adapt to patients’ preferences as these may have limited engagement and enrollment among marginalized patient populations. Because inclusiveness and equity are important elements of programmatic success, future programs will need flexible and multi-channel human-to-human communication pathways for complex clinical interactions or patients who do not desire tech-first approaches.


COVID Watch Overview
Patients were enrolled in COVID Watch by outpatient or ED clinicians through an application embedded in Penn Medicine's electronic health record. Patients were additionally offered COVID Watch via an automated text message if they received a positive result from a COVID-19 test conducted by a Penn Medicine laboratory.
Once enrolled, COVID Watch sent twice daily text messages in English or Spanish that asked "How are you feeling compared to 12 hours ago: better, same, or worse?" Patients who replied "worse" were subsequently asked, "Is it harder than usual for you to breathe: yes or no?". Patients who responded "yes" generated an electronic health record (EHR) alert, monitored by a team of telemedicine clinicians (nurses, advanced practice providers, and physicians) available 24/7, who contacted the patient within one hour.
Interpretation services were used when needed. Patients could also text the word "worse" at any time to connect with a clinician, also within one hour. During the study period, some participants were automatically randomized to receive a home ngertip pulse oximeter or not. 10 Those who received a pulse oximeter were asked to report their symptoms and their ambulatory oxygen saturation levels. Patients whose oxygen saturation, SpO 2 , was below 90% or had decreased by more than 3% and below 95%, were called by the clinical staff who triaged the patient using standardized protocols. The program was free and did not require a patient to have established care with a Penn Medicine provider (e.g., primary care) or insurance. We describe the COVID Watch interventions, including the pulse oximeter trial, in previous publications. 7,10,16 Study Participants And Setting We conducted semi-structured interviews with three groups of people who interacted with COVID Watch: (a) patients who had been enrolled in COVID Watch within the prior 90 days, (b) primary care and ED clinicians who directly enrolled patients in COVID Watch or had their patients enrolled into COVID Watch, and (c) administrators in primary care or the ED. Ethical approval was provided by the Institutional Review Board at the University of Pennsylvania.

Patient Recruitment And Sampling Strategy
Patients were recruited via phone between February and June 2021, a timespan that included the randomized controlled trial of ngertip pulse oximetry among COVID Watch enrolled patients. 10 Patients were purposively sampled across two patient-level strata to gain a diversity of patient perspectives on their COVID Watch experience: (a) having a pulse oximetry device mailed to their house (or not), and (b) level of engagement in the program (high vs. low, de ned as responding to twice-daily, automated text message prompts at least 10 out of 14 days of enrollment versus fewer). The cutoffs were based on the median level of engagement. We monitored and recruited patients to attempt balance across racial, ethnic, and language sub-groups. Patients provided verbal informed consent were compensated $50 USD (United States dollar) for their time.

Clinician And Administrator Recruitment And Sampling Strategy
Clinicians (e.g., physicians, nurse practitioners, and physician assistants) and administrators (e.g., medical directors, physician leads, or non-clinical practice managers) were recruited using emails between July and November 2021. Clinicians and administrators were purposively sampled across two health system level strata in hopes of gaining a diversity of provider perspectives of COVID Watch: (a) primarily worked in an ED setting vs primary care setting, and (b) providers that enrolled a high versus low number of patients in COVID Watch. For both settings, high enrolling clinicians were de ned as those who enrolled ≥ 15 patients and low enrollers were de ned as less than 10 patients. These cutoffs were based on the median level of enrollment. Administrators were recruited from the clinical sites of the providers.
Snowballing techniques were used to identify additional clinicians or staff who might have been in uential in encouraging clinicians or staff members to enroll patients in COVID Watch. Clinicians and administrators provided verbal informed consent and were compensated $50 USD for their time.
Interviews were conducted before the publication of COVID Watch's evaluation. 7,10 Interview Guide Development We created three semi-structured, open-ended interview guides (see Table 1, Supplementary 1, Supplementary 2, Supplementary 3) for each cohort: patients, providers, and administrators. Guides were created by core members of the research team (KHC, JS, MW, MN, JR, MKD, AF, SK, MG, AM) and reviewed by the larger team. They were pilot tested with at least two participants in each cohort. Questions were open-ended and included follow-up probes to allow participants to expand upon answers. After the interview, participants self-reported sociodemographics. Transcripts were then entered into NVivo 1.5 (QSR International) for coding and analysis.
Separately for patients, clinicians, and administrators, early interview transcripts were used to develop an initial codebook using a modi ed content analysis 17,18 approach that relied upon the structure of the interview guide but allowed for emergent themes. The codebook was applied to all transcripts. Recruitment and emergent ndings were reviewed in bi-weekly team meetings. The achieved inter-rater reliability was ĸ=0.81 across co-coded transcripts. Patient interviews lasted an average of 53 minutes (range 37-74 minutes) and clinician and administrator interviews lasted an average of 36 minutes (range 23-54 minutes).
All study protocols and instruments were deemed exempt by the University of Pennsylvania's Institutional Review Board.

Participant Characteristics
In total, 85 interviews were completed. Forty-seven patients were interviewed, who were on average 50 years old and mostly female, White, non-Hispanic, and English speaking (see Table 1). Because most administrators were clinicians and no major thematic differences were identi ed between clinicians and administrators, we combine these groups for presentation and refer to both groups as "clinicians" henceforth. The sample of 38 clinicians was primarily female, White, non-Hispanic, and physicians, and had been in practice for 11 years or more (see Table 2). Across all cohorts, themes aligned into three categories: (1) sentiments about COVID Watch, (2) feedback for improving COVID Watch and only asked of clinicians, (3) lessons learned from COVID Watch that have implications for future remote patient monitoring programs (see Table 3). There was no notable thematic difference by level of patient engagement or clinicians' predilection (high vs. low) for enrolling patients into COVID Watch, therefore themes are aggregated across strata. Results speci c to patients with pulse oximetry devices and clinicians' practice setting (ED vs. primary care) are noted below. Quotes to illustrate each theme are presented in Table 4 for patients and Table 5 for clinicians.     Irritating Some participants felt that the text messages were excessive, intrusive, or annoying. These feelings were common for patients at the ends of the illness spectrum, either with mild to no symptoms, or conversely so ill they did not have the energy to respond to messages in a timely manner. Those who had a low response rate to the twice daily messages were often participants who felt the text messages were unhelpful.

Insu cient
Some patients who were worried about their COVID-19 symptoms or those who were fearful about the risk of severe illness expressed a desire for real-time support from human clinicians instead of the automated, routine text messages provided by COVID Watch. Some participants wanted to report additional symptoms beyond feeling short of breath.

Clinician Perspectives
Comforting ED clinicians often described taking comfort in knowing that COVID Watch could monitor patients discharged home. Many described it as a valuable "safety net" for patients especially when COVID-19 was a new illness. This feeling of a safety net was particularly true in clinicians' discussions of patients with signi cant social needs (e.g., without a primary care provider, socially isolated) and patients who did not meet admission criteria but whom they worried might decompensate at home after being discharged from the ED.
ED and primary care clinicians believed that COVID Watch also gave their patients a sense of comfort. They knew the program would monitor them and provided an alternative to the ED as the sole source of COVID-19 care.
Some clinicians saw the provision of pulse oximetry devices to use at home as an important way to give providers and patients additional data points about the severity of a patient's illness. For example, one ED clinician described being able to trust a patient's report of dyspnea more if the patient used a pulse oximeter at home.
ED clinicians did not believe that the ability to enroll patients in the program in uenced their decisions to admit versus discharge.
Increased Access to Care for Patients COVID Watch's accessibility was seen as a key bene t to the program. Patients had quick and easy access to a clinician if needed; patients were not alone in their health decision-making; the program alleviated patient fear of the unknown; and the program was free of charge. Additionally, a few clinicians reported COVID Watch increased access for their Spanish-speaking patients, which was described as a key need at some sites.

Reduced Follow-up Burden
Many primary care clinicians found COVID Watch to be an important tool for managing follow-up care when the volume of patients' needs was high.
Feedback For Improving Covid Watch

Patient Perspectives
Improve the Enrollment Process Some patients did not recall when or how they were enrolled in COVID Watch. Patients who knew they were enrolled tended to describe more positive feelings about starting the program.
A misconception about the program was that some patients thought it was their own doctor who had enrolled them in COVID Watch and was personally monitoring their symptoms.

Clarify the Monitoring and Escalation Process
Patients desired clearer, more concise information about what symptoms would result in clinical escalations and found the subjective nature of the daily text message (e.g., "Are your symptoms the same, better, or worse than 12 hours ago") to be challenging. Some patients desired more quantitative measures such as a 0-10 number scale for their dyspnea. This desire for quantitative measures was also re ected in patients' positive reception to the pulse oximetry device. The device provided an objective measure that enabled patients to feel more con dent about their clinical course.
Some patients expressed a preference for phone calls over text messages, referencing the di culty that older patients can have with texting, or not having phones that are equipped for text messaging. Other patients thought phone calls would be preferable because they would give clinicians more clarity about how patients are feeling.
Spanish-speaking patients more often felt COVID Watch was not able to fully meet their needs. Some Spanish speakers expressed how Spanish-speaking cultures tended to be more phone-call oriented, and so an option to choose the modality of the messages may provide a better cultural t for some.

Clinician Perspectives
Improve the Enrollment Process Clinicians tended to describe the process of enrolling into COVID Watch as relatively easy, but there was a desire to make enrollment even easier. Some believed it was tedious to, for example, go into the patient's exam room to ensure patients received the program's initial text message or to ensure the patient's phone number was correct. In addition, some clinicians felt that their own familiarity with the enrollment process waned if they had not enrolled a patient recently. Some also expressed uncertainty about the program's details such as the ability to enroll patients over the weekends.

Provide Solutions for Patients with Limited Device Access or Hesitancy
Clinicians highlighted barriers related to patients' accessing COVID Watch and hesitancy to use their phones for engaging in care. Participation required the ability to use a cell phone with text messaging.
Access to the required technology was particularly challenging for elderly patients (especially those who lived alone) and patients experiencing homelessness.

Address Low-Literacy and Language Preferences Among Patients
A key access-related barrier clinicians discussed was that the program required comfort in reading and writing in English or Spanish; other languages should be considered. Other enhancements for accessibility included offering an option for patients to use a landline; offering access to a central hotline phone number those patients could call; or distributing cell phones for patients to use. Finally, some clinicians recommended that patients be given the ability to enroll themselves.

Create a Feedback Loop for Clinicians
ED and primary care clinicians discussed a desire to know the clinical course of their patients after enrollment in COVID Watch and were interested in knowing which patients did not escalate, those who escalated to a COVID Watch nurse, or those who unenrolled from the program early. This would serve as a mechanism to inform clinicians about the quality of their care, particularly if enrolling in COVID Watch was effective, and a reminder that the COVID Watch program was still enrolling patients.

Lessons For Future Of Remote Patient Monitoring Programs From Clinicians Enhanced Data Collection
Clinicians felt that remote patient monitoring programs will be an important part of practicing medicine in the future. However, data collected should have concrete benchmarks. Clinicians were hesitant about COVID Watch's subjective self-reports and felt more objective measures should be used in future programs, COVID-19 related or otherwise.
Both ED and primary care clinicians felt remote patient monitoring programs should provide patients with the appropriate health data collection tools, like a home pulse oximeter or blood pressure cuff, to collect and report data back to their healthcare team. Some also suggested greater integration with existing electronic health records, directly embedding remotely recorded results into the medical record.
A Guide for Patients ED clinicians felt that remote patient monitoring could reduce the number of ED visits by giving patients more accurate, objective data about when not to come to the ED. By using objective data and clear cutoffs, patients could be clearly guided to seek the right level of care. In addition, the ability to provide reassurance to patients with a remote monitoring program might help patients being discharged home from the ED.

Extend Remote Patient Monitoring to Non-COVID-19 Conditions
Many clinicians also felt remote patient monitoring will be particularly valuable for certain chronic and acute conditions. For example, participants tended to perceive targeted data collection to be practical and effective for monitoring and treating conditions like congestive heart failure, diabetes, asthma, weight management, and post-surgery recovery. To evolve for other use cases, however, clinicians emphasized that remote patient monitoring tools needed to be easy to use and equitable for patients, and clinicians must be con dent in the quality of the data collected.

Discussion
Overall, while patients and clinicians found COVID Watch to be comforting and bene cial, improvements to the design and implementation of the program will be important for the program's future and have implications for the design of future remote patient monitoring programs.
COVID Watch often provided a sense of comfort and reassurance to both patients and clinicians, facilitated by increased access to care because of its free cost and Spanish-language availability. Despite their enthusiasm, patients and clinicians desired a better user experience. Patients wanted more clarity about when and how they were enrolled in the program. The confusion over enrollment may have contributed to lower engagement in the program, and also may have lowered perceptions of its utility. Patients and clinicians also desired the program to build in exibility, expressing concerns about the universal use of automated text messages for patients. For example, the program may have had limited uptake among patients who had reticence about using technology to communicate their health needs, limited English or Spanish literacy, or preferred additional languages. These subgroups are among those who are at increased risk for limited access to care and experienced worse COVID-19 outcomes. 19 Our ndings have generated three key insights for future remote patient monitoring programs to manage COVID-19 or other clinical conditions. First, remote patient monitoring programs should not be static, onetime builds or implementations. While these programs may have automation or use digitized algorithms, they are human facing programs that should evolve as technology advances, patient and clinician expectations of technology evolve, and standards for managing targeted disease conditions change.
Developing systems for monitoring program performance and patient engagement and seeking patient and clinician feedback to continuously re ne these programs are as important as achieving intended health outcomes.
Second, health systems must acknowledge the human resources needed to support remote patient monitoring programs, even if automation is embedded in the program. While automation can improve e ciency for some patients, successful programs will need to marry technology with options for human interaction. Tech-rst approaches may not always be welcomed. Some patients in our study indicated the desire to connect with a human clinician, wanting to avoid the automated text message system more generally. Yet, at the same time, automation allowed for a team of 3-4 nurses to simultaneously manage e ciently and effectively over one thousand patients during the sta ng shortages of the pandemic, 7 a challenge projected to remain over the coming years. 20,21 Using default pathways (e.g., text messages) that are automated, complemented by alternative pathways (e.g., interactive voice recordings or humanto-human phone calls) that are customized to the user's needs, may be one solution for greater engagement while not overburdening current clinicians.
Finally, future programs must be designed with equity as a primary principle, recognizing patients who have the most limited access to care may need additional design considerations. For example, programs should be offered in multiple languages. In our study both patients and clinicians expressed concerns that vulnerable populations may have been excluded because more direct human-to-human connections (e.g., telephone calls) were not made available. Developing programs that correctly balance patients' desires for human-to-human connections, promoting inclusivity, with the e ciency gains of automated processes will be important for future remote patient monitoring programs.
This study has limitations. Our analysis took place in one large academic institution. However, the institution includes six hospitals and over 500 outpatient practices across a wide geographic area, allowing us to sample participants from multiple hospitals, encompassing urban and suburban settings across the large catchment area. These interviews took place relatively early in the pandemic -within the rst year -and therefore re ect the stress that both patients and clinicians felt when faced with an unprecedented crisis. Finally, patients' and clinicians' experiences with COVID Watch and COVID evolved during the study period such as the implementation of automated and opt-out enrollment in the fall of 2020. In addition, surges of infection, the increased availability of vaccination, and effective treatments might have in uenced our participants' responses.
Remote patient monitoring programs are a growing form of health care delivery and are being tested in a variety of clinical conditions including hypertension management, in-home administration of chemotherapy, and transitions between hospital and home. Success of these programs hinges on usercentered design to enhance experiences for both patients and clinicians 22 as well as intentional design for traditionally marginalized groups who have not historically been considered as early adopters of new technology-based care programs. [23][24][25] Understanding how a diverse group of patients engage with and experience remote patient monitoring programs, and how providers integrate them into daily work ow and clinical decision-making will be informative for future remote patient monitoring programs. The datasets generated and/or analyzed during the current study are not publicly available due to the researchers not compiling or analyzing any datasets as part of this qualitative research, but additional information from the corresponding author is on reasonable request.

Competing interests
None.

Funding
This work was supported primarily by a grant from the Patient Centered Outcomes Research Institute (COVID-2020C2-10830-Delgado, PI). The authors were also supported by the National Institutes of Health (Delgado -K23HD090272001, Chaiyachati -K08AG065444) and with a philanthropic grant from the Abramson Family Foundation (Delgado) during the study period.

Role of the Funding Source
The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.