Feasibility of electronic patient-reported outcome monitoring with self-management advice and warning system in aplastic anemia and paroxysmal nocturnal hemoglobinuria – a pilot study (ePRO-AA-PNH)

Introduction: Electronic patient-reported outcome (ePRO) systems have demonstrated to improve patient outcome in cancer, but have not yet been evaluated in patients with aplastic anemia (AA) and paroxysmal nocturnal hemoglobinuria (PNH). This pilot study evaluated the feasibility of an ePRO monitoring, self-management advice and warning system customized for AA and PNH patients. Methods: After adapting an ePRO system to the needs of AA and PNH patients (ePRO-AA-PNH), the application was tested by patients and their medical team for 6 months. Based on a predened algorithm symptom reporting triggered self-management advice and prompts to contact clinicians in case of severe symptoms. Meanwhile the medical team could monitor ePROs in real-time and receive alerts of severe symptoms electronically. Results: Nine patients completed the study, showing a high adherence rate to the weekly symptom-reporting (72%), which decreased from 91 % to 53 % over the study period. Satisfaction was high: ePRO-AA-PNH was found easy to use, to understand and to integrate into daily life. Most patients (78%) would continue and all would recommend the application to other AA/PNH patients. Technical performance was rarely a barrier and healthcare providers saw ePRO-AA-PNH as a useful supplement, but missed the integration to clinical workows and the hospital electronic health records (EHR). Conclusion: An ePRO system customized for AA and PNH was feasible to patients and their medical team in terms of adherence, satisfaction and performance, showing a high potential for these rare and chronic conditions. However, the integration into clinical workows and hospital EHR is crucial for routine use.


Introduction
Patient reported outcomes (PRO) are de ned as any report of the status of a patient's health condition that comes directly from the patient, without interpretation by a clinician. PROs can be gathered electronically (ePRO), which offers several potential advantages over paper-based PROs and has shown to be feasible with high levels of compliance (1)(2)(3)(4)(5)(6). Electronic PRO systems cannot only collect data on a regular basis and ag symptoms which health providers regularly miss or downgrade (7), but also can send out automatic noti cations directly to patients or clinicians about alarming symptoms. In cancer patients this approach has repeatedly shown to enhance patient-clinician communication and to facilitate rapid clinical intervention, resulting in less emergency visits, improved overall survival and health-related quality of life (HR-QoL) (8)(9)(10) .
Based on this extensive experience of ePRO systems in cancer, aplastic anemia (AA) and paroxysmal nocturnal hemoglobinuria (PNH) patients are ideal candidates for such ePRO interventions. AA and PNH are rare chronic conditions, characterized by symptoms of bone marrow failure such as fatigue, bleeding and infections (11)(12)(13) in AA and by complications of hemolysis and thrombosis in PNH (14,15). Despite major improvements in therapies in the past decades, AA and PNH patients often remain symptomatic with signi cantly reduced HR-QoL as well as requiring longterm followup due to their risk for disease relapse, severe complications and late effects (16)(17)(18)(19)(20).
In this setting symptom-monitoring with ePROs may have the potential to detect symptoms and life-threatening complications earlier, thereby improving disease management. Since both entities occur in young adults, this patient group may be particularly accepting with respect to the use of digital tools.
Finally, in these rare conditions ePROs systems are a vital opportunity to guide patients remotely when living a greater distance from their specialized medical center. To date, an ePRO system has neither been adapted to the needs of this patient group nor been tested in AA/PNH patients.
The aim of this pilot study was to develop a disease-speci c ePRO system for routine symptom monitoring with self-management advice and warning system, which later was examined for feasibility in AA and/or PNH patients.

Study design and study population
This single center study followed a mixed-methods design using qualitative and quantitative methods to test the feasibility of an ePRO system adapted to the needs of AA and PNH patients. Between November and December 2019 nine patients with AA and/or PNH and their corresponding medical team were prospectively enrolled in the study at the University Hospital Basel. Patients were eligible if they were older than 18 years and had internet access with prior email experience. Patients with mental alteration or psychiatric disease, which would compromise informed consent or study adherence, as well as patients, who were not able to read or write German were excluded. Participation in the study was offered by the treating physician and after the patient's approval to participate, the patients were approached by the study team by telephone, email or during routine visits for study inclusion. The study was approved by the local ethics committee (EKNZ Nr. 2019 − 01563) and registered under ClinicalTrials.gov (NCT04128943). All parts of the study were performed within the principles of the Declaration of Helsinki.
Disease-speci c ePRO system customization The ePRO system provided by Kaiku Health® (Helsinki, Finland) was utilized for symptom monitoring. This browser-based platform has been used in cancer patients before (21,22) and is accessible by computers, tablets and smartphones with access to the internet.
For this study, the system was adapted from its original form to the speci c needs of AA and PNH patients (development overview in supplemental material, Fig. 3) since previous studies have shown the necessity for a user-centered redesign of health information technology (23)(24)(25). For symptom monitoring two disease-speci c PRO questionnaires previously developed by our group based on a literature review and expert consensus rounds were used. These questionnaires analyse 11 core symptoms for AA or PNH (26): Fatigue, fever, bleeding, dyspnea, pain, mood swings and concentration de cits was assessed in both entities; palpitations, tremors, muscle cramps and paresthesia in AA and hemoglobinuria, jaundice, dysphagia and erectile dysfunction in PNH. Each symptom question was subdivided into 3 domains: occurrence (yes/no), severity (mild, moderate, severe, very severe) and interference with daily activity (HR-Qol). Both questionnaires included one openended question to detect additional symptoms and allow comments. For each symptom we integrated symptom-management advice based on internal institution recommendations and international guidelines, which patients would receive upon reporting symptoms. During an implementation phase, two AA patients, two nurses and two physicians experienced in the care of AA and PNH were given full access to the Kaiku® prototype for several days.
Thereafter, semi-structured qualitative interviews were performed to assess opportunities for improvement. Based on the feedback from these interviews the Kaiku® system was further optimized, and in particular functions for documenting vital parameters, current blood counts and timepoint of eculizumab infusion were added. Data extraction of these interviews are listed in the supplemental material (Table 3).

Clinical test phase
After giving informed consent, patients were instructed by the study investigator on how to use the application (duration: 30 minutes). All patients were assigned to a symptom monitoring program depending on their diagnosis and sex (AA, PNH female, PNH male), which consisted of a weekly symptom questionnaire and an email reminder. Previous work has identi ed an optimal recall time of seven days (27). In addition to the questionnaire, the patients were asked to measure and document their vital parameters weekly (blood pressure, pulse and temperature), to ll out a Quality of Life questionnaire in the rst and last study week, to report their current blood counts and if applicable the timepoint of eculizumab infusion within the application.
The ePRO system incorporated a prede ned algorithm to stratify patient-reported symptoms in real-time into minor, moderate and severe symptoms (26), each resulting in different actions (Tables 1 and 4, Fig. 4): In case of minor or moderate symptoms, patients would automatically receive selfmanagement advice. Severe symptoms would generate emergency rst aid instructions and a warning to consult their medical team or, if not available, the emergency room.
Two dedicated physicians of the research team reviewed patient responses within the application daily and reported severe symptoms to the responsible physicians at the treatment center. Upon study initiation all involved physicians and nurses were invited to create a Kaiku® login and were allowed to use the application for their daily routine work. It was left to the responsible physician on how to further manage the care of the patient with respect to reported symptoms.
Participants tested the application for a total of six months. During this period patient interviews by telephone were conducted at three weeks and at three months. Upon study completion after 6 months, patients as well as the involved nurses and physicians evaluated the application in a nal interview.

Data analysis
In this feasibility study we assessed recruitment, adherence, user experience and technical performance of an ePRO system. PRO questionnaire adherence rates were quanti ed by calculating the proportions of completed questionnaires compared to the observation period in weeks.
Questionnaires lled in addition to the weekly assessments were not counted to the adherence rate. Based on recent implementation guidelines (28) we conducted semi-structured interviews for the assessment of usability, e ciency, user satisfaction and technical issues of the application and evaluated the results using thematic analysis. Frequency and severity of patient-reported symptoms were summarized descriptively. Quantitative statistical analyses were performed using R version 4.01 (R core team, Vienna, AT).

Patient characteristics and recruitment
During November and December 2019, 14 patients were screened and asked to participate in the study. Four patients did not want to participate due to a lack of motivation (n=4, 29%, median age: 27 years IQR 26-30) and one patient was excluded due to limited German language skills (n=1, 7%). Nine patients were enrolled, resulting in a recruitment rate of 64%. Median age was 35 years (IQR 29-56), whereby more females (n=6, 67%) than males (n=3, 33%) were included in the study. Five patients were diagnosed with AA, three with PNH and one with overlapping AA/PNH. Median disease duration was 9 years (IQR 6-13). Further patient characteristics are listed in Table 2.

Patient adherence
A total of 234 weekly reminders were sent and 168 questionnaires completed, resulting in a 72% questionnaire adherence rate. Figure 1 illustrates the monthly adherence rate, showing a decreasing rate from 91% (month 1) to 53% (month 6) over the total study period. One patient completed all questionnaires over the 6-month study period. Besides the weekly questionnaires, an additional questionnaire was lled out in 12 instances. Figure 2 demonstrates all reported symptoms for AA and PNH respectively. A total of 331 symptoms were reported, of which 154 (47%) were classi ed as mild, 95 (29%) as moderate and 82 (25%) as severe. The most common reported symptom was fatigue in AA (44 entries) and in PNH (43 entries).

Symptom reporting and triggered actions
Symptoms of severe grade were most often reported as bleeding in AA (14 entries) and fatigue in PNH (21 entries). Eighty-two symptoms were graded as severe according to the prede ned algorithm, 28 due to symptom severity, 23 due to interference with daily activity and 31 due to both reasons equally. Severe symptoms led to 36 alerts being sent out to the treating physicians. These alerts did not lead to additional physician contacts or admissions of the patients, as patients either managed the symptoms directly at home or presented themselves directly to their medical care team already before reporting these symptoms within the application. In addition to the prede ned symptom questions, additional symptoms reported were nausea, cough, itching and chest pressure. Technical problems were reported in 3 cases, all of which were due to di culties with a password reset (two nurses and one physician). Patients did not report any technical problems or concerns on data security. The application was mostly accessed by smartphone (n = 7, 78%). Overall, the usability of the application was rated as "easy to use" while the symptom questionnaire was classi ed as "easy to understand" by all patients.
One patient mentioned di culties in grading the severity of symptoms ("I had trouble deciding if my pain was minor or moderate"). Of the ve patients who reported severe symptoms, two stated that the warnings and recommendations to contact a physician were in line with their personal experience and symptom management. Three patients reported that the warnings were only sometimes in agreement with their personal experience, of which all three stated that the warning for "fatigue" was triggered too soon. Four patients never triggered an automatic warning.
Reasons for non-adherence were forgetting (n=4, 44%), being too busy (n=4, 44%), the lack of any symptoms (n= 1, 11%) and not wanting to be reminded of the disease (n=1, 11%). While some patients (n = 3, 33%) deemed the self-instructions for symptom care useful, almost half of the patients (n = 4, 44%) regarded the self-instructions as "not very useful" since they were already trained in disease management after having lived with the disease for many years (median duration 9 years). They suggested that less experienced and newly diagnosed patients would bene t the most from this application. All patients rated the time expenditure as "appropriate". While 6 patients (67%) would like to continue to use the app weekly, 2 patients (22%) would prefer to use it less often (every two weeks), and 1 patient (11%) more often (every 3 days). The integration of the app into daily life was rated as "very easy" by all patients.
Overall, all patients reported to be satis ed with the application. While 7 patients (78%) would continue to use the app, 2 patients (22%) would stop using it. All patients would recommend the application to other AA or PNH patients. Recommendations for improvement focused on the automatic integration of vital parameters, blood values and medication into the app, a calendar function for medication and hospital visits, a possibility to ask the care-team questions and voluntary social functions.
All members of the medical team described the application as "easy to use". While more than half of the medical team members (n = 4, 57%) reported a bene t from the tool, others (n = 3, 43%) did not. Both nurses felt con dent in their ability to screen and manage the reported symptoms. Though not everybody saw a personal bene t from the tool, all caretakers would continue to use the application due to the likely bene t provided to patients. All recommended that the tool should be integrated into the electronic hospital records and clinical work ow. To improve data interpretation, it was suggested to integrate data on therapies into the application. Another suggestion was an alarm function for patients directly integrated into the start page of the application. Data extraction from the interviews can be found in the supplemental materials (

Discussion
This pilot study con rmed that electronic symptom-monitoring including a warning system is feasible for both patients and their clinicians in AA and PNH. To our knowledge this is the rst study that assessed an ePRO systems in AA and PNH, as previous ePRO trials predominantly focussed on cancer patients (1,8,9). Patient interest and recruitment rate for the ePRO system was high, suggesting that the approach can be used broadly for this rare and chronic condition. We did not see an effect of age on recruitment or adherence rate as other studies have shown before (1, 10), although our small study population does not allow general assumptions. Overall, patients adherence to the weekly symptom questionnaire was high (72%), but decreased over time with the highest rates at the beginning (91%) and the lowest rates after 6 months (53%). Impaired functional status or technical issues were rarely a barrier, instead the main reason for reduced engagement was that patients forgot and for the medical team the insu cient integration into their clinical work ows and hospital EHR, which is in line to other ePRO studies (1,5) This study was performed as add-on to routine work ows, so neither medical staff nor patients saw it as part of their core disease management. Consequently, in order for ePROs to become part of routine care it is pivotal to overcome organizational and technological barriers for ePROs (29,30). This also holds true for the collection of vital parameters, blood counts and therapies, which were documented infrequently in our study and underlines the need for a direct interface with the hospital information system for automatic synchronization. For successful implementation of ePRO systems in clinical practice, enough time and resources need to be allocated to IT integration and ongoing support. It is also crucial to de ne responsibility, time resources and management during non-o ce hours of the medical team (31,32).
In contrast to previous studies in cancer patients (8,9), the warning system, which sent out alerts to the patients in case of severe symptoms, did not result in any additional hospital admissions. Since it is a key aim of ePRO systems to guide patients to contact their medial team on time before symptoms worsen and cause downstream complications, we further investigated the reasons for this surprising nding. Some experienced patients stated that they managed symptoms already on their own or called their physicians before lling out the ePRO questionnaire, re ecting the long disease history of many of the included patients (median: 9 years). However, this does not fully explain why none of the patients actually followed the alert to contact the medical team. In particular for the most common reported symptom "fatigue", the patients may have shown signs of resignation as they had already previously learned of the sometimes limited therapeutic options (13). Second, fatigue is a highly complex symptom, which is di cult to assess by a simple question and thus may have triggered potentially false alarms. This reveals a major drawback of short ePRO questionnaires (10-12 items). These can be completed quickly (max. 15 min) and therefore are commonly used within ePRO systems due to their high practicability, however they potentially miss-out on important details needed to assess complex symptoms. In AA and PNH it might be of additional value to re ne instruments for certain symptoms such as, "fatigue" -as one of the most common symptoms often persisting despite adequate therapy -or "neurocognitive de cits". Unfortunately, such speci c instruments for assessing these symptoms in AA/PNH are not available, although for cancer-related fatigue (e.g. The Functional Assessment of Chronic Illness Therapy -Fatigue Scale (33)) already well established. This emphasizes the need for the further development of disease-speci c PRO questionnaires in AA/PNH.
Next to the symptom monitoring and warning system, the inclusion of symptom-management advice based on a prede ned algorithm for the severity was a key part of the ePRO system. Interestingly, it was only helpful for some patients, whereas others saw less bene t. This might be explained by the fact that we mainly included patients advanced in their disease course. Consequently, this tool might have a bigger impact on AA/PNH patients in their early disease journey as they learn to better manage their symptoms.
Although this feasibility study was not designed to assess the frequency of reported symptoms of AA/PNH patients, and therefore these results should be interpreted with caution, we still can make out interesting observations. Using our recently developed PRO questionnaire (26) it was possible to detect less known symptoms in this disease entity (e.g. concentration problems), which could point towards an underestimation of these symptoms in AA/PNH and opens a window for future research. However, higher patient numbers are needed to reliably assess the symptom burden in AA/PNH with the help of ePRO.
Overall, the user experience was very positive. Most patients and clinicians would continue to use the application and the medical team regarded the tool as a useful addition to routine management. For inexperienced physicians the tool might be especially helpful to estimate the severity and monitor the disease course, as it is di cult to gain extensive clinical experience in these rare diseases. The multidimensional recording of symptoms, blood values and therapies could also form the basis for an automatic prediction of adverse events and disease course as has been attempted in other elds (34,35), in particular when considering the future bene ts of arti cial intelligence and machine learning.
With the focus on the rare occurrence of AA and PNH, several needs of this ePRO system for the future could be identi ed: A calendar function with reminders for questionnaire completion, medication intake and hospital visits may also improve drug compliance (36,37), which increasingly gets important as emerging therapies can be administered at home (e.g. oral and subcutaneous complement inhibitors). Also a social function could provide patients a valuable opportunity to create a network, share their experiences, and improve patient engagement while far away from their specialized medical centers. Considering the current COVID-19 pandemic, this function is also of great value to directly communicate with the care team in a convenient and remote way as recently showed successful in cancer patients (38). Besides the utility in patient care, this disease-speci c ePRO system is also useful in collecting data for future trials seeking real-world information on patient symptom burden and quality of life.
Several limitations of the study merit consideration. First, the small patient sample may have overestimated adherence rates compared to a broader patient sample and cannot represent patient demographics in general. In addition, a larger patient population with patients at different disease and treatment stages is necessary to detect further missing symptoms of the ePRO questionnaire. However, in rare diseases large patient groups are di cult to recruit.

Conclusion
An ePRO system customized for AA and PNH was feasible to patients and their medical team in terms of adherence, satisfaction and performance, showing a high potential for these rare and chronic conditions. However, the integration into clinical work ows and hospital EHR is crucial for routine use. The results can be seen as a valuable resource to further improve ePRO tools for this patient group, but also serve as an impetus to translate such patient-centered work ows to other hematological diseases and routine care in hematology.

Declarations Ethics approval
The study was approved by the local ethics committee (EKNZ Nr. 2019-01563) and registered under ClinicalTrials.gov (NCT04128943). All parts of the study were performed within the principles of the Declaration of Helsinki.

Consent for publication
All participants have given consent for publishing their clinical data in anonymized form.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Authors' disclosures of potential con icts of interest RA is an employee of Kaiku Health Ltd. All other authors indicated no potential con icts of interest.

Funding
This study received funding from a junior research grant of the University Basel, a grant of the ProPatient foundation, unrestricted grants from Kaiku Ò Health Ltd. and Novartis AG. Patient adherence to the weekly questionnaire over the 6 month study time.