Outcome domains and measurement instruments of patient- relevant improvement of structure and processes as a new set of outcomes for evaluating and approving digital health applications: systematic review

doi:10.21203/rs.3.rs-2872706/v1

Background: In October 2020, digital health applications (DiGAs) became part of standard care in Germany. For approval, DiGA manufacturers must demonstrate medical benefit or patient-relevant improvement of structure and processes (PISP). PISP refers to an innovative outcome core area in terms of proof of benefits and reimbursement decisions. These are subdivided into 9 outcome domains, including health literacy, facilitating access to care, and coping with illness-related difficulties in everyday life. Their implementation aims at empowering patients, encouraging shared decision-making, and increasing patient-centeredness in healthcare delivery. Given the novelty of PISP, no standardized set of outcomes and outcome measurement instruments currently exists to operationalize the domains. Learning from previous evaluation studies can help operationalize and standardize PISPs for evaluation studies of digital health applications.

Therefore, we investigated the outcomes and outcome measurement instruments, used in controlled trials to assess DiGA-compliant applications, published before the Digital Health Applications Ordinance of April 2020.

Methods: We conducted a systematic review of studies published between 01/2015 and 04/2020, via MEDLINE and Embase, complemented by forward/backward searches. Controlled trials assessing interventions adhering to the definition of DiGA were eligible, if they applied a validated outcome measurement instrument, and if results were presented in German or English. Title-abstract screening, full-text screening, data extraction and qualitative synthesis were conducted independently by two researchers.

Results: Out of 2,671 references identified, 6 studies collecting a total of 48 outcomes were included. 14 outcomes (29.2%) addressed PISP by using 13 different measurement instruments. The outcomes corresponded to 5 of 9 PISP outcome domains with health literacy beeing the most common (7/14, 50.0%).

Conclusions: This review provides an overview of the characteristics of PISPs used in previous evaluation studies of DiGA-compliant applications. It shows which outcomes and validated outcome measurement instruments can be used to measure PISP and where knowledge is still lacking. These results serve as a starting point for operationalizing and standardizing PISPs and help to increase the outcome measurement quality of PISPs.

Digital health application

patient-relevant improvement of structure and processes

digital health

mhealth

ehealth

telemedicine

outcome and process assessment

health care

patient reported outcome measures

health services research

Since October 2020, digital health applications (so-called DiGAs) have been part of standard care for people covered by statutory health insurance in Germany. Given Germany’s pioneering role in international comparison, other countries and health systems follow with great interest its implemented coverage policies and assessment processes for digital health applications.(1, 2) The legal and regulatory requirements were set by the Digital Healthcare Act (Digitale-Versorgung-Gesetz; DVG) from December 2019 and the Digital Health Applications Ordinance (Digitale-Gesundheitsanwendungen-Verordnung; DiGAV) from April 2020.

Definition of DiGA

DiGAs are active, lower-risk medical devices, classified as class I or IIa medical devices according to the European Union (EU) Medical Device Regulation 2017/745 (MDR).(3) Therefore, a medical purpose to be achieved by the digital main function of the DiGA must be defined by the manufacturer. This function is intended to support the recognition, monitoring, treatment, or alleviation of diseases or the recognition, compensation, treatment, or alleviation of injuries or disabilities. The target group for DiGAs is patients with a confirmed diagnosis, which must be assignable to a 3- or 4-digit ICD-10 code.(4) All necessary features of a DiGA can also be found in the DiGA Guide for Manufacturers, Service Providers and Users (5) in accordance with Section 139e of Book V of the Social Security Code (Sozialgesetzbuch V) and are summarized in Figure 1. DiGAs can be used with the prescription of treating physicians and psychotherapists or after patients request their use and approval directly from their statutory health insurance.

At the request of the manufacturer and as a precondition for reimbursement, the applications are reviewed, assessed, and approved by the Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte; BfArM).(5) In addition to the required characteristics of DiGAs outlined above, the BfArM also reviews whether the manufacturer has provided proof that their application fulfills basic requirements such as patient safety, functionality, data protection, data security, quality, and interoperability of the medical device.(5) If the manufacturer has provided proof that these basic requirements are fulfilled, the BfArM will review whether the manufacturer has demonstrated a positive healthcare effect of their application. If all these requirements are met, the DiGA will be approved and included in the DiGA directory maintained by the BfArM.(6) Listing in the DiGA directory is a prerequisite for the prescription of a DiGA and reimbursement of its costs by the statutory health insurance.

Evaluation of positive healthcare effects

Positive healthcare effects can be demonstrated by proof of medical benefit or patient-relevant improvement of structure and processes (PISP). The term medical benefit refers to outcomes that are known from trials of clinical or pharmaceutical interventions, including (a) improvement of the state of health, (b) reduction of the duration of a disease, (c) prolongation of survival, or (d) improvement in the quality of life. In contrast, PISP refers to an outcome core area in the context of proof of benefits and in terms of reimbursement, innovative in Germany and by international comparison (see figure 2).(1) The 9 PISP outcome domains are (a) coordination of treatment procedures, (b) alignment of treatment with guidelines and recognized standards, (c) adherence, (d) facilitating access to care, (e) patient safety, (f) health literacy, (g) patient autonomy, (h) coping with illness-related difficulties in everyday life, and (i) reduction of therapy-related efforts and strains for patients and their relatives.(5) Table 2 describes each outcome domain, as defined by the DiGA Guide of the BfARM.(5)

In the context of assessment and approval, positive healthcare effects in terms of both medical benefits and PISPs are now equally important. Therefore, for only one of these a positive healthcare effect must be demonstrated compared to standard care.(5) The motivation behind this decision was to empower patients to become more active and informed, encourage shared decision-making, and promote health literacy.(5) In addition, the integration of this new outcome core area reflects a more comprehensive understanding of the quality of care and its evaluation for the purpose of patient benefits. Therefore, it increases patient-centeredness in healthcare delivery and contributes to the principles of value-based healthcare.(7, 8) Furthermore, this approval process and the coverage of costs by statutory health insurances provide the basis for patients having low-threshold access to quality-assured digital health applications.(5)

The perspective of patients on their health status and healthcare delivery, and their participation in their therapy, have become increasingly relevant in evidence-based medicine.(9-11) This is reflected by the significant increase in the use of patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs)(12-14) that can also be observed regarding digital health interventions.(10, 15) In particular, all 9 PISP outcome domains can be assigned to PROMs. PROMs are used to monitor health conditions and the effectiveness of treatments and interventions, whereas PREMs are used to evaluate and monitor experiences during the delivery and use of healthcare services, both measured from the patient perspective.

Research gaps concerning PISP

Patient perspective is key in the various evaluation frameworks and guidelines on digital health interventions. However, they differ in their definition and operationalization of the patient perspective. For example, the authors of the Model for the Assessment of Telemedicine Applications (MAST) recommend that the patient perspective should be considered by measuring usability and acceptance.(16) The National Institute for Health and Care Excellence (NICE) framework proposes within evidence tier C, which relates to applications that fit the definition of a DiGA, that the focus should be on measuring effectiveness in terms of quality of life or symptom severity.(17) Frameworks also exist that recommend the measurement of particular PISP outcome domains as part of the evaluation of digital health applications, such as the Khoja–Durrani–Scott Evaluation Framework (18) and the design and evaluation framework for digital health interventions (DEDHI) (19), which covers aspects such as “improved access to care,” “equity of care,” “effects on the delivery of care,” and “service quality.”

Thus, some elements of the PISP domains are considered by existing frameworks for evaluating health applications. However, to date, none cover all 9 domains. Moreover, given the novelty of PISPs as outcome domains for the this evaluation, no standardized set of outcomes exists to operationalize the 9 domains, let alone a set of measurement instruments to assess them. Still, as demonstrated by the aforementioned frameworks, some of the aspects that define the 9 PISP domains have been considered important for evaluating health applications. Therefore, learning from previous evaluation studies can help standardize outcomes and measurement instruments for the collection of PISP in the course of evaluation studies of digital health applications. For medicial benefits of analogue as well as digital interventions, comprehensive measurement tools exist, some of which are codified within Core Outcome Sets.(20)

Against this background, this study aimed to examine the characteristics of PISP measured in previous prospective controlled evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV.

Research questions

This systematic review examined the following research questions:

What were the characteristics of evaluation studies in which PISPs were collected?
Which outcome domains, outcomes, and outcome measurement instruments were used to assess PISP and medical benefits?
How frequently were the different outcome domains, outcomes, and outcome measurement instruments used in the included evaluation studies?
What was the ratio of outcomes from the PISP core area to outcomes from the medical benefit core area across the included studies?

Search strategy

To investigate these research questions, we conducted an in-depth analysis of the evaluation studies of telemedicine applications included in the systematic review of Knapp et al. published in November 2021.(15) The review investigated the use of PROMs and PREMs in those evaluation studies. Since PISPs belong to the field of PROMs as outlined above, this review provided an appropriate basis to investigate our research questions. The preceding review was performed as an electronic database search on MEDLINE and Embase. The inclusion and exclusion criteria, as well as the search string can be found in Appendix 1. The search string was based on previous works of other research groups on the topics of telemedicine (21) and PROMs (22). The screening of the 2,671 hits in the databases, the extraction as well as the data analysis were each performed independently by two reviewers.

The initial review included 303 studies providing the basis for our in-depth analysis. Studies up to April 2020 were included in the review because the Digital Health Applications Ordinance became effective then and we aimed to examine evaluation studies published before it. In addition, we performed an in-depth hand search, as well as a comprehensive forward and backward reference search starting from the research items finally included in the original review in order to ensure that we considered all relevant articles in our review. Back and forward as well as hand searches were finalized in September 2021.

Eligibility criteria

We defined inclusion and exclusion criteria for the in-depth analysis presented in the current paper in order to identify evaluation studies for telemedicine applications complying with the definition of a DiGA in accordance with the requirements mentioned within the introduction. The inclusion and exclusion criteria were based on participant, intervention, comparison, outcome, and study type (PICOS scheme) as shown in Table 1.

Study selection and data extraction

Based on the eligibility criteria, two researchers (MS, AK) independently conducted a title and abstract screening of all articles included in the initial review as well as those gained from hand, forward and backward searches. They then independently assessed the full texts of the preselected articles for inclusion. Any disagreements were discussed between both reviewers and resolved in discussion.

Data extraction was conducted by both researchers (MS, AK) independently. To ensure a consistent approach to data extraction, the matrix for data extraction was jointly developed in advance. The selection of relevant characteristics for the extraction matrix was based on characteristics commonly used in systematic reviews, including author, title, year of publication, journal, study country, study type, and intended medical purpose indicated by 3-digit ICD-10. These were supplemented by characteristics relevant to answering our research questions, including (1) main intended use of the DiGA, describing the type of study intervention; (2) assignment of outcome(s) to the categories of medical benefit, patient-relevant improvement of structure and processes, or others; (3) outcome domain(s); (4) outcome(s); and (5) outcome measurement instrument(s). The latter were divided into validated and non-validated instruments, considering that the DiGAV requires the use of validated outcome measurement instruments to demonstrate the healthcare effects of digital health applications. Figure 2 illustrates this subdivision, based on examples from the initial review by Knapp et al.(15)

For the purpose of characterizing the main intended use of the DiGAs, we used the terminology in the DiGA Guide, which in turn matches the MDR.(3) The main intended uses included recognition, monitoring, treatment, alleviation, and compensation.

To assign outcomes to the respective PISPs, we used the DIGA Guide as a reference due to a lack of existing binding definitions of the different PISPs so far.(5) Table 2 includes all PISP domains and corresponding explanations as direct citations translated by the authors from the German original DIGA guide.

Please insert Table 2: Description of PISP outcome domains based on the DIGA Guide (5)

The results of the independent data extraction were discussed by both reviewers (MS, AK). Any disagreements were discussed and resolved by consensus. Appendix 2 shows the entire data extraction table.

As our review explicitly does not aim to prove the effectiveness of DiGAs, but rather to focus on the allocation and application of different outcome domains for the evaluation of DiGAs, we did not perform a risk of bias analysis which would have been necessary in the context of an effectiveness analysis.

Study selection

The initial review by Knapp et al. identified 2,671 studies, which resulted in 303 included studies. After applying the inclusion and exclusion criteria of the in-depth analysis presented here, 133 studies remained for title and abstract screening, and 17 were included in full-text screening. Six studies met all inclusion criteria and were included in data extraction. 11 studies were excluded, because no PISPs were measured as outcome domains of the intervention (n=6)(23-28), PISPs were not measured by validated outcome measurement instruments as required in the approval process for DiGAs in Germany (n=3)(29-31), a non-controlled study design was used (n=2)(27, 32) or it was a study protocol (n=1)(33). The flow chart in Figure 3 shows the entire study selection process. Hand, backward and foward searches did not yield any additional studies which met all inclusion criteria.

Study characteristics

Of the 6 included studies, 4 were randomized controlled trials (34-37), 1 was a controlled pragmatic pilot trial (38), and 1 was a controlled trial (39). The studies were published between 2016 and 2019 and covered a study period from 2009 to 2017. Three studies were conducted in the United States (35, 37, 39), and 1 was conducted in each of Sweden (38), India (34), and Canada (36). The indications addressed by the DiGA-compliant telemedicine applications were chronic obstructive pulmonary disease (COPD)(38), type 2 diabetes mellitus (34), osteoarthritis (35), bronchial asthma (36), chronic heart failure, and spinal cord injury (37). Of the included studies, five investigated a monitoring intervention (34-37, 39), and one investigated an intervention for alleviating symptom burden (38). Further characteristics of the included studies can be found in Appendix 2 but are not shown here because they were not relevant to answer our research questions.

Characteristics of outcome core areas, outcome domains, outcomes, and outcome measurement instruments

A total of 48 outcomes were collected in the 6 included studies, contributing to a mean of 8.0 outcomes per study. Of these, 14 (29.2%) outcomes addressed PISP, and 29 (60.4%) addressed medical benefit. The remaining 5 (8.3%) outcomes could not be assigned to the core areas of PISP or medical benefit and hence were assigned to the category “other.” They included outcome domains such as satisfaction and usability.

PISPs are divided into 9 outcome domains as outlined in the introduction. The 14 outcomes identified in the studies of our review could be assigned to 5 of these domains. The most commonly used PISP outcome domain was health literacy, with a frequency of 7 (50.0%), followed by coping with illness-related difficulties in everyday life, with a frequency of 3 (21.0%) and adherence with a frequency of 2 (14.0%) (Table 3). Within our studies, we found no outcomes belonging to the domains coordination of treatment procedures, facilitating access to care, patient autonomy, or patient safety.

A total of 13 different measurement instruments were used for the assessment of PISP outcome domains. One outcome measurement instrument, the PROM Patient Activation Measure (PAM)-13 questionnaire(40), was used twice. Table 3 and Appendix 2 provide a comprehensive overview of all PISP and medical benefit outcome domains, outcomes, and outcome measurement instruments used, presented according to the individual studies.

Please insert Table 3: Overview of included studies detailing PISP and medical benefit outcomes, outcome priority, and outcome measurement instruments

The majority of PISP outcomes (71.4%, 10/14) were assessed by measurement instruments in the form of validated questionnaires, as shown in Figure 4. One questionnaire (7.1%) was self-developed for evaluation purposes in a single study. In 3 cases (21.4%), PISPs were not collected by questionnaires but by patient-reported data such as self-reported medication or frequency of blood glucose testing according to schedule. In comparison, outcomes belonging to the core area of medical benefit were assessed less frequently by validated questionnaires (55.2%, 16/29).

All of the 6 included studies investigated outcomes from the core area of PISP in addition to outcomes from the core area of medical benefits. We found no study investigating solely outcomes from the core area of PISP. Furthermore, outcomes from the core area of PISP were investigated as secondary outcomes across all included studies except the one by Evangelista et al., where no difference for primary and secondary endpoints was made.(39)

Table 3 provides a comprehensive overview of all included studies detailing the investigated outcome domains, the outcome including their priority, and the applied outcome measurement instruments.

Our study aimed to examine the characteristics of PISP use in the context of previous prospective evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV in April 2020.

Main findings in the context of previous research

The core area of PISP was introduced to strengthen the role of patients and take their assessments and benefits into greater account in the approval of digital health applications (6). Notably, PISP outcome categories are primarily process quality indicators and not outcome quality indicators as represented by medical benefit. Considering process quality indicators in addition to outcome quality indicators, which mutually influence each other, provides the basis for a more holistic, not to mention patient-centered, evaluation of digital health applications.(63)

Altogether, we included 6 studies in our review. All included studies used a controlled study design, 4 (4/6) of which were randomized controlled trials. The approval process as a DiGA in Germany also requires a controlled study design, underlining the appropriateness of this design for demonstrating the effectiveness of digital interventions compared to non-controlled study designs.(64-66) The majority of evaluated applications focused on patients with chronic conditions (4/6) and offered monitoring of the corresponding disease as a feature (5/6). Both findings reflect the current range of telemedicine applications, which primarily offer monitoring for patients with chronic conditions.(15)

The most commonly used PISP outcome domain in evaluation studies was health literacy (7/14, 50.0%), followed by coping with illness-related difficulties in everyday life (3/14, 21.0%). One possible reason for this is that health literacy (67) is a widespread and well-known outcome domain and various established outcome measurement instruments already exist in the form of validated questionnaires. For instance, the Health Literacy Tool Shed online database listed a total of 240 health literacy measurement instruments as of April 28^th 2023. (68)

Within our studies, we found no outcomes belonging to the PISP domains of patient autonomy, coordination of treatment procedures, facilitating access to care, or patient safety. Since validated measurement instruments for most of these domains exist (69-71), and increasing patient empowerment and access to care are among of the key promises of digital health use (72), this result is rather surprising.

Regarding the PISP outcome domains in our evaluation studies, self-developed questionnaires or process-generated, tracked, or directly measured data were used for only 4 out of 14 (28.5%) of the outcomes measured. In this regard, our study provides evidence that for some PISP outcome categories, validated outcome measurement instruments may be lacking. This is also critical because validated questionnaires are mandatory in the DiGA evaluation studies for measuring and demonstrating a positive healthcare effect.(5)

In addition, the analysis showed that clearly assigning the outcomes from evaluation studies to the PISP outcome domains is sometimes difficult. This is due to a lack of detailed guidance on which outcomes can be assigned to each domain and which outcome measurement instruments should be used for assessment.(5)

Notably, all included studies evaluated outcomes from the core area of PISPs in addition to outcomes from the core area of medical benefits and only as secondary outcomes. No study solely evaluated positive effects for outcomes from the core area of PISPs. This might be due to the novelty of PISP outcome domains in the context of approval trials and the fact that positive healthcare effects in terms of medical benefits and PISPs are now of equal importance.

Implications for future research

Our study is the first systematic review of evidence concerning the characteristics of PISPs in the context of previous evaluation studies of DiGA-compliant digital health applications. Therefore, our results are a starting point concerning the guidance needed on which outcomes and outcome measurement instruments can be used in evaluation studies of such applications to measure PISP outcome domains. The results also highlight PISP outcome domains where knowledge is still lacking about outcomes and outcome measurement instruments that can be used. Thus, our findings can be used to further develop existing evaluation frameworks and outcome taxonomies. The discussion of our findings also provides several valuable ideas for targeted implementation and dissemination of PISP in Germany, as well as other countries that would like to strengthen the patient perspective in the evaluation and implementation of digital health applications.

Future research should advance the distinct and transparent assignment of outcomes and validated outcome measurement instruments to all 9 of the existing PISP categories based on the results of our review. Research is especially needed concerning the 6 domains for which we found no insights concerning validated outcome measurement instruments.

Another topic for future research is the further inclusion of PISP outcome categories in existing evaluation frameworks and outcome taxonomies. The questions of how PISPs can be classified within existing taxonomies, how taxonomies could be adapted, or even if new ones should be developed hold comprehensive research potential.

Additionally, updating the review in one or two years will be of great interest as it will help to analyze how the characteristics of PISP measurement in evaluation studies have developed since the first DiGA was approved in Germany in October 2020.

Implications for practice

With DiGAs now being part of standard care for people with statutory health insurance in Germany and with PISPs now relevant in terms of approval and reimbursement decisions, Germany is breaking new ground in international comparison. These innovations, the accompanying discussions, and thus, the results of the present study are of international interest for different target groups:

(I) international digital health application manufacturers and vendors who plan to have their application approved as a DiGA in Germany, an important market in the field of mobile health applications [2]

(II) representatives of governments, and healthcare systems worldwide concerning the significance of PISP in evaluation studies of digital health applications and the associated DiGA approval and reimbursement process [2]

(III) patients, who gain low-threshold access to quality-assured and evidence-based digital health applications and whose perspective and potential benefits have become significantly more important through innovative German legislation. This greater consideration of patient benefits when implementing and evaluating care interventions follows the principles of value-based healthcare [7].

Limitations

Currently, no legally binding document elaborates on the single PISP outcome categories, the corresponding outcome domains, outcomes, or outcome measurement instruments. This lack of explanation made it difficult for us to assign the single outcomes to the respective PISP outcome categories. However, to sharpen our understanding, we used the explanations from the DiGA Guide (Table 2)(5) and insights from a text book published by an expert committee of the Federal Ministry of Health for digital health interventions.(73) Furthermore, the interventions and study settings described in the articles were considered for the assignment of outcomes to the respective PISP outcome categories, and the entire data extraction was done by two reviewers independently. Nonetheless, the difficulties we faced illustrate again the need for further clarification of which outcomes and outcome measurement instruments can be assigned to the 9 PISP outcome domains.

As this was the follow-up analysis of an existing review not directed at measuring the effectiveness of an intervention, no a priori registration of the review was filed. However, the previous review used validated search strategies as well as a piloted data extraction strategy.

Given the fact that we did not aim to measure intervention effectiveness, we refrained from any quality assessment. Therefore, no statement can be made on potential risks of bias within the included studies.

Since we were unable to obtain any additional references from the extensive aditional searches, we assume that the initial data set and thus the conduct of a subreview was an appropriate method to address our research questions.

Our review provides an overview of the characteristics of PISP use in the context of previous prospective controlled evaluation studies of DiGA-compliant digital health applications, published internationally before the DiGAV in April 2020. Our findings also show which outcomes and outcome measurement instruments can be used in evaluation studies of digital health applications to measure PISP outcome domains. Concurrently, we have highlighted PISP outcome domains where knowledge is still lacking about outcomes and validated outcome measurement instruments that can be used. Given the few studies included, the results are a starting point for operationalizing and standardizing PISPs and, therefore, increase the outcome measurement quality of PISPs.

The possibility of demonstrating positive healthcare effects by proof of PISP, and that this is now relevant for cost reimbursement decisions by statutory health insurance funds, underlines the active role and personal responsibility of patients in their health. This should become even more important in terms of patient-centered care. Prescription and care practice will show to what degree DiGAs and outcomes from the PISP core area are presumed relevant by healthcare providers and patients alike. The outlined need for a clear assignment of outcomes to individual PISP outcome categories is also pivotal to enable correctly informed decision-making by physicians and patients for or against a DiGA.

BfArM – Federal Institute for Drugs and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte)

DiGAV – Digital Health Applications Ordinance (Digitale-Gesundheitsanwendungen-Verordnung)

DVG – Digital Healthcare Act (Digitale-Versorgung-Gesetz)

MDR – Medical Device Regulation 2017/745

NICE – National Institute for Health and Care Excellence (NICE)

PISP – Patient-relevant improvement of structure and processes

PROMs – Patient-reported outcome measures

PREMs – Patient-reported experience measures

Ethics approval and consent to participate

not applicable

Consent for publication

not applicable

Availability of data and materials

The detailed search strategy and complete data extraction sheet can be found as additional files.

Competing interests

MS, AK and LH declare that there is no conflict of interest. JS received institutional grants for investigator-initiated trials from Novartis, Sanofi, ALK, and Pfizer. He acted as a consultant for Novartis, ALK, Lilly, and Sanofi.

Funding

The review was part of the project “Häusliche Gesundheitsstation,” funded by the European Regional Development Fund and the Free State of Saxony (06/2018-09/2022; Grant number: 100278533).

Authors’ contribution

MS and AK jointly planned, conducted, and analyzed the review and drafted the manuscript. LH supported the conceptualization of the manuscript and reviewed and revised the first draft. JS obtained funding for the project and supported the conceptualization and realization of the review. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Acknowledgements

The authors would like to thank the European Regional Development Fund and the Free State of Saxony for making this study possible. We would also like to thank Stefanie Deckert for supporting the review with her expertise and project-independent perspective.

Lauer W, Löbker W, Höfgen B. Digitale Gesundheitsanwendungen (DiGA): Bewertung der Erstattungsfähigkeit mittels DiGA-Fast-Track-Verfahrens im Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM). Bundesgesundheitsbl. 2021;64(10):1232-40.
Gerke S, Stern AD, Minssen T. Germany’s digital health reforms in the COVID-19 era: lessons and opportunities for other countries. npj Digit Med. 2020;3:94.
The European Parliament and the Council of the European Union. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. 2017. https://eur-lex.europa.eu/eli/reg/2017/745/oj. Accessed Apr 28, 2023.
World Health Organization. ICD-10: International Statistical Classification of Diseases and Related Health Problems: 10th Revision. Geneva: World Health Organization; 2011.
Federal Institute for Drugs and Medical Devices (BfArM). The fast-track process for digital health applications (DiGA) according to section 139e SGBV. A guide for manufacturers, service providers and users. Bonn: Federal Institute for Drugs and Medical Devices (BfArM); 2020. https://www.bfarm.de/SharedDocs/Downloads/ EN/MedicalDevices/DiGA_Guide.pdf. Accessed Apr 28, 2023.
Federal Institute for Drugs and Medical Devices (BfArM). DiGA-Verzeichnis. 2023. https://diga.bfarm.de/de/verzeichnis. Accessed Apr 28, 2023.
Porter ME. What is value in health care. N Engl J Med. 2010;363:2477-81.
Teisberg E, Wallace S, O'Hara S. Defining and Implementing Value-Based Health Care: A Strategic Framework. Acad Med. 2020;95:682-5.
Schlieter H. Timpel P, Otto L., Richter P., Wollschlaeger B., Knapp A., Harst L. Digitale Gesundheitsanwendungen – Forderungen für deren Entwicklung, Implementierung und begleitende Evaluation. Monitor Versorgungsforschung. 2021;14:76-9.
European Commission. Assessing the impact of digital transformation of health services. Luxembourg: Publications Office of the European Union; 2019. https://ec.europa.eu/health/sites/default/files/expert_panel/docs/022_digitaltransformation_en.pdf. Accessed Apr 28, 2023
McGee RG. How to Include Patient-Reported Outcome Measures in Clinical Trials. Current Osteoporosis Reports. 2020;18:480-5.
Kargo AS, Coulter A, Jensen PT, Steffensen KD. Proactive use of PROMs in ovarian cancer survivors: a systematic review. J Ovarian Res. 2019;12:63.
Siljander MP, McQuivey KS, Fahs AM, Galasso LA, Serdahely KJ, Karadsheh MS. Current Trends in Patient-Reported Outcome Measures in Total Joint Arthroplasty: A Study of 4 Major Orthopaedic Journals. J Arthroplasty. 2018;33:3416-21.
Weszl M, Rencz F, Brodszky V. Is the trend of increasing use of patient-reported outcome measures in medical device studies the sign of shift towards value-based purchasing in Europe? Eur J Health Econ. 2019;20 Suppl 1:133-40.
Knapp A, Harst L, Hager S, Schmitt J, Scheibe M. Use of Patient-reported Outcome Measures and Patient-reported Experience Measures within Evaluation Studies of Telemedicine Applications: Systematic Review. J Med Internet Res. 2021;23:e30042.
Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, et al. A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 2012;28:44-51.
National Institute for Health and Care Excellence (NICE). Evidence standards framework for digital health technologies. 2018. https://www.nice.org.uk/corporate/ecd7/resources/evidence-standards-framework-for-digital-health-technologies-pdf-1124017457605. Accessed Apr. 28 2023
Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed J E Health. 2013;19:48-53.
Kowatsch T, Otto L, Harperink S, Cotti A, Schlieter H. A design and evaluation framework for digital health interventions. it - Inf Technology. 2019;61:253-63.
COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments). 2023. https://www.cosmin.nl/. Accessed Apr 28, 2023.
Arnold K, Scheibe M, Müller O, Schmitt J, und die CCS THOS Konsensgruppe. [Principles for the evaluation of telemedicine applications: Results of a systematic review and consensus process]. Z Evid Fortbild Qual Gesundhwes. 2016;117:9-19.
Mackintosh A, Comabella C, Hadi M, Gibbons E, Roberts N, Fitzpatrick R. PROM group construct and instrument type filters. Department of Public Health, University of Oxford, Oxford. 2010. https://cosmin.nl/wp-content/uploads/prom-search-filter-oxford-2010.pdf. Accessed Apr 28, 2023.
Kumar D, Tully LM, Iosif AM, Zakskorn LN, Nye KE, Zia A, et al. A Mobile Health Platform for Clinical Monitoring in Early Psychosis: Implementation in Community-Based Outpatient Early Psychosis Care. JMIR Ment Health. 2018;5:e15.
van der Meij E, Anema JR, Leclercq WKG, Bongers MY, Consten ECJ, Schraffordt Koops SE, et al. Personalised perioperative care by e-health after intermediate-grade abdominal surgery: a multicentre, single-blind, randomised, placebo-controlled trial. Lancet. 2018;392:51-9.
Merchant R, Inamdar R, Henderson K, Barrett M, Su JG, Riley J, et al. Digital Health Intervention for Asthma: Patient-Reported Value and Usability. JMIR Mhealth Uhealth. 2018;6:e133.
Zairina E, Abramson MJ, McDonald CF, Li J, Dharmasiri T, Stewart K, et al. Telehealth to improve asthma control in pregnancy: A randomized controlled trial. Respirology. 2016;21:867-74.
Tran C, Dicker A, Leiby B, Gressen E, Williams N, Jim H. Utilizing Digital Health to Collect Electronic Patient-Reported Outcomes in Prostate Cancer: Single-Arm Pilot Trial. J Med Internet Res. 2020;22:e12689.
Kubo A, Kurtovich E, McGinnis M, Aghaee S, Altschuler A, Quesenberry C, Jr., et al. A Randomized Controlled Trial of mHealth Mindfulness Intervention for Cancer Patients and Informal Cancer Caregivers: A Feasibility Study Within an Integrated Health Care Delivery System. Integr Cancer Ther. 2019;18:1534735419850634.
Wills AM, Garry J, Hubbard J, Mezoian T, Breen CT, Ortiz-Miller C, et al. Nutritional counseling with or without mobile health technology: a randomized open-label standard-of-care-controlled trial in ALS. BMC Neurol. 2019;19:104.
Ramkumar PN, Haeberle HS, Ramanathan D, Cantrell WA, Navarro SM, Mont MA, et al. Remote Patient Monitoring Using Mobile Health for Total Knee Arthroplasty: Validation of a Wearable and Machine Learning-Based Surveillance Platform. J Arthroplasty. 2019;34:2253-9.
McElroy I, Sareh S, Zhu A, Miranda G, Wu H, Nguyen M, et al. Use of digital health kits to reduce readmission after cardiac surgery. J Surg Res. 2016;204:1-7.
Le Marne FA, Butler S, Beavis E, Gill D, Bye AME. EpApp: Development and evaluation of a smartphone/tablet app for adolescents with epilepsy. J Clin Neurosci. 2018;50:214-20.
Lidington E, McGrath SE, Noble J, Stanway S, Lucas A, Mohammed K, et al. Evaluating a digital tool for supporting breast cancer patients: a randomized controlled trial protocol (ADAPT). Trials. 2020;21:86.
Kleinman NJ, Shah A, Shah S, Phatak S, Viswanathan V. Improved Medication Adherence and Frequency of Blood Glucose Self-Testing Using an m-Health Platform Versus Usual Care in a Multisite Randomized Clinical Trial Among People with Type 2 Diabetes in India. Telemed J E Health. 2017;23:733-40.
Skrepnik N, Spitzer A, Altman R, Hoekstra J, Stewart J, Toselli R. Assessing the Impact of a Novel Smartphone Application Compared With Standard Follow-Up on Mobility of Patients With Knee Osteoarthritis Following Treatment With Hylan G-F 20: A Randomized Controlled Trial. JMIR Mhealth Uhealth. 2017;5:e64.
Ahmed S, Ernst P, Bartlett SJ, Valois MF, Zaihra T, Paré G, et al. The Effectiveness of Web-Based Asthma Self-Management System, My Asthma Portal (MAP): A Pilot Randomized Controlled Trial. J Med Internet Res. 2016;18:e313.
Kryger MA, Crytzer TM, Fairman A, Quinby EJ, Karavolis M, Pramana G, et al. The Effect of the Interactive Mobile Health and Rehabilitation System on Health and Psychosocial Outcomes in Spinal Cord Injury: Randomized Controlled Trial. J Med Internet Res. 2019;21:e14305.
Nyberg A, Tistad M, Wadell K. Can the COPD web be used to promote self-management in patients with COPD in swedish primary care: a controlled pragmatic pilot trial with 3 month- and 12 month follow-up. Scand J Prim Health Care. 2019;37:69-82.
Evangelista LS, Lee JA, Moore AA, Motie M, Ghasemzadeh H, Sarrafzadeh M, et al. Examining the effects of remote monitoring systems on activation, self-care, and quality of life in older patients with chronic heart failure. J Cardiovasc Nurs. 2015;30:51-7.
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004;39:1005-26.
Riegel B, Lee CS, Dickson VV, Carlson B. An update on the self-care of heart failure index. J Cardiovasc Nurs. 2009;24:485-97.
Rector TS, Johnson G, Dunkman WB, Daniels G, Farrell L, Henrick A, et al. Evaluation by patients with heart failure of the effects of enalapril compared with hydralazine plus isosorbide dinitrate on quality of life. V-HeFT II. The V-HeFT VA Cooperative Studies Group. Circulation. 1993;87 6 Suppl:Vi71-7.
Lorig KR, Sobel DS, Ritter PL, Laurent D, Hobbs M. Effect of a self-management program on patients with chronic disease. Eff Clin Pract. 2001;4:256-62.
Juniper EF, Guyatt GH, Cox FM, Ferrie PJ, King DR. Development and validation of the Mini Asthma Quality of Life Questionnaire. Eur Respir J. 1999;14:32-8.
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606-13.
Brooks R. EuroQol: the current state of play. Health Policy. 1996;37:53-72.
Ehrs PO, Nokela M, Ställberg B, Hjemdahl P, Wikström Jonsson E. Brief questionnaires for patient-reported outcomes in asthma: validation and usefulness in a primary care setting. Chest. 2006;129:925-32.
Emilsson M, Berndtsson I, Lötvall J, Millqvist E, Lundgren J, Johansson A, et al. The influence of personality traits and beliefs about medicines on adherence to asthma treatment. Prim Care Respir J. 2011;20:141-7.
Stern RA, Arruda JE, Hooper CR, Wolfner GD, Morey CE. Visual analogue mood scales to measure internal mood state in neurologically impaired patients: Description and initial validity evidence. Aphasiology. 1997;11:59-71.
Buran CF, Brei TJ, Sawin KJ, Stevens S, Neufeld J. Further development of the adolescent self management and independence scale: AMIS II. Cerebrospinal Fluid Research. 2006;3:S37.
Hall K, Dijkers M, Whiteneck G, Brooks CA, Krause JS. The Craig Handicap Assessment and Reporting Technique (CHART): Metric Properties and Scoring. Top Spinal Cord Inj Rehabil. 1998;4:16-30.
Law M, Baptiste S, McColl M, Opzoomer A, Polatajko H, Pollock N. The Canadian occupational performance measure: an outcome measure for occupational therapy. Can J Occup Ther. 1990;57:82-7.
Beck AT, Steer RA, Brown G. Beck depression inventory–II. Psychological Assessment. 1996.
Skevington SM, Lotfy M, O'Connell KA. The World Health Organization's WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual Life Res. 2004;13:299-310.
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and validation of the Patient Assessment of Chronic Illness Care (PACIC). Med Care. 2005;43:436-44.
Kroll T, Kehn M, Ho P-S, Groah S. The SCI Exercise Self-Efficacy Scale (ESES): development and psychometric properties. Int J Behav Nutr Phys Act. 2007;4:34-.
Wångdahl JM, Mårtensson LI. The communicative and critical health literacy scale--Swedish version. Scand J Public Health. 2014;42:25-31.
Jones PW. COPD assessment test --rationale, development, validation and performance. Copd. 2013;10:269-71.
Bestall JC, Paul EA, Garrod R, Garnham R, Jones PW, Wedzicha JA. Usefulness of the Medical Research Council (MRC) dyspnoea scale as a measure of disability in patients with chronic obstructive pulmonary disease. Thorax. 1999;54:581-6.
EuroQol--a new facility for the measurement of health-related quality of life. Health Policy. 1990;16:199-208.
Frändin K, Grimby G. Assessment of physical activity, fitness and performance in 76-year-olds. Scand J Med Sci Sports. 1994;4:41-6.
Olsson SJ, Ekblom Ö, Andersson E, Börjesson M, Kallings LV. Categorical answer modes provide superior validity to open answers when asking for level of physical activity: A cross-sectional study. Scand J Public Health. 2016;44:70-6.
Donabedian A. The definition of quality: a conceptual exploration. The definition of quality and approaches to its assessment. Health Administration Press. 1980:3-31.
Sachverständigenrat zur Begutachtung der Entwicklung im Gesundheitswesen. Digitalisierung für Gesundheit. Ziele und Rahmenbedingungen eines dynamisch lernenden Gesundheitssystems. 2021. https://www.svr-gesundheit.de/fileadmin/Gutachten/Gutachten_2021/SVR_Gutachten_2021.pdf Accessed Apr 28, 2023.
Peiris D, Miranda JJ, Mohr DC. Going beyond killer apps: building a better mHealth evidence base. BMJ Global Health. 2018;3:e000676.
Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating Digital Health Interventions: Key Questions and Approaches. Am J Prev Med. 2016;51:843-51.
Tavakolikia N, Kheiltash A, Shojaeefar E, Montazeri A, Shariati M, Meysamie A. The most well-known health literacy questionnaires: a narrative review. Social Determinants of Health. 2017;3:104-13.
Health Literacy Tool Shed: A database of health literacy measures. https://healthliteracy.bu.edu/. Accessed Apr 28, 2023.
Singla AK, Kitch BT, Weissman JS, Campbell EG. Assessing Patient Safety Culture: A Review and Synthesis of the Measurement Tools. Journal of Patient Safety. 2006;2:105-15.
Clement S, Brohan E, Jeffery D, Henderson C, Hatch SL, Thornicroft G. Development and psychometric properties the Barriers to Access to Care Evaluation scale (BACE) related to people with mental ill health. BMC Psychiatry. 2012;12:36.
Kasper J, Hoffmann F, Heesen C, Köpke S, Geiger F. MAPPIN'SDM – The Multifocal Approach to Sharing in Shared Decision Making. PLOS ONE. 2012;7:e34849.
Eysenbach G. What is e-health? J Med Internet Res. 2001;3:e20.
Brönneke JB, Debatin JF, Hagen J, Kircher P, Matthies H. DiGA VADEMECUM: Was man zu Digitalen Gesundheitsanwendungen wissen muss: Berlin: Medizinisch Wissenschaftliche Verlagsgesellschaft; 2020.

Table 1: Inclusion and exclusion criteria

	Inclusion criteria	Exclusion criteria
Patients	All patient groups
Intervention	Treatment with a DiGA-compliant telemedicine application	Treatment with a non-DiGA-compliant telemedicine application
Control	Non-digital standard care	DiGA vs. DiGA
Outcome	Outcome domain(s) measured by at least one validated outcome measurement instrument covering PISP (the availability of a validation study served as an indicator for a validated outcome measurement instrument, following the definition in the initial review (15))	Outcome domain(s) not measured by at least one validated outcome measurement instrument covering PISP
Studies	Randomized controlled trials Controlled trials Publications in English or German Publication date between 2015 and April 2020	Papers, guidelines, and handbooks about DiGAs in general Reviews Case reports Non-controlled trials Feasibility studies Retrospective studies Qualitative studies Study protocols Publications not in English or German

Table 2: Description of PISP outcome domains based on the DIGA Guide (5)

Term	Explanation
Adherence	“Adherence refers to the implementation of parts of the patient’s therapy that have been agreed between patient and physician, or parts of the therapy necessary within the framework of treatment in accordance with guidelines. Adherence thus requires the cooperation of the patient and underlines their active role in the implementation of a therapy. DiGA can support in fulfilling this active role, such as by enabling better integration of health behavior and everyday activities. The relevance of the improvements that can be achieved in this way is enormous; in international studies, 30–50% of chronically ill patients on long-term medication are not adherent to the agreed therapeutic measures.”
Alignment of treatment with guidelines and recognized standards	“Guidelines and other recognized treatment standards not only cover the actions of healthcare providers but also describe how the patient can or must contribute to the success of the therapy. For various diseases such as diabetes mellitus, for example, patient guidelines not only make the guidelines originally for physicians easy for laypersons to understand but also explain what patients themselves can do. DiGA can translate such instructions into concrete formats that are suitable for everyday use, be individually adapted, and help to ensure that treatment is based on guidelines and other recognized standards throughout, i.e., even when the patient is not with the physician. For example, this could be by reminding patients of necessary visits to the physician, explaining and motivating them to perform regular exercises at home, or supporting them in achieving a sustainable change in their lifestyle.”
Coordination of treatment procedures	“DiGA can support the coordination of treatment processes between one or more healthcare providers on the one hand and the patient on the other. Improvements in care can result, such as from a therapy that is particularly well-adapted to the acute support needs of the patient, a better-organized therapy process, or communication possibilities that are low-threshold and event-related.”
Coping with illness-related difficulties in everyday life	“DiGA can support patients in reducing and coping with everyday illness-related difficulties. For example, digital health applications can use sensor technology or data evaluation to warn of seizures at an early stage or detect an imminent increase in symptoms so that patients can better prepare for them. They can facilitate care and monitoring by relatives, e.g., allowing monitoring a patient even at a distance, and they can help to develop individual strategies for dealing with a disease to enable better social participation.”
Facilitating access to care	“Similar to telemedicine services, DiGA can help to improve patients’ access to care and support equal and reliable access to health services, regardless of place of residence and other factors.”
Health literacy	“According to the majority of the population in Germany, finding, understanding, correctly classifying, evaluating, and using health-related information is difficult. Health literacy is important to be able to make decisions in everyday life that facilitate maintaining health or support the success of a therapy. In the context of a therapy, DiGA can provide patients with relevant health information that is important for their actions and support them in understanding and implementing the therapy through individualized performance adapted to the needs of the target group to strengthen and ensure its success.”
Patient autonomy	“Patients are important contributors to their own health. Their experience and knowledge hold great potential for improving all areas of the healthcare system, which must be used. Through patient orientation and participation, the prevention of diseases and the health status and quality of life of patients can be improved. DiGA can enable and strengthen patients’ autonomous health behavior and effectively support their involvement in decision-making processes concerning their health.”
Patient safety	“Patient safety is a priority objective of healthcare and a guiding principle for the further development of the health system. The extensive quality and safety specifications developed to reduce risks and avoid treatment can be strengthened by DiGA and extended from the events in clinics and medical practices to the patient’s home environment. They can enable patients to recognize and even react to increased risks in a treatment, errors in the application of a therapy, or undesirable individual effects. For example, DiGA with a medication management function can effectively support patients in safe drug therapy.”
Reduction of therapy-related efforts and strains for patients and their relatives	“DiGA can organize the treatment procedures and daily handling of disease more effectively for patients and their relatives. This saves time and effort and reduces avoidable physical or psychological stress for those involved. Examples are simplification of measurements and recordings, support in deciding whether a visit to the physician is necessary (such as through the correct classification of side effects), and strengthening the feeling of safety under therapy. In addition, data analyses can be used to plan visits to the physician in a more targeted manner if necessary.”

Table 3: Overview of included studies detailing PISP and medical benefit outcomes, outcome priority, and outcome measurement instruments

Study	Study type	Intervention	Outcome core area	Related PISP outcome domain	Outcome	Out-come priority	Outcome measurement instrument
Evangelista et al. 2015 (39)	Controlled trial	Remote monitoring web intervention for adults with heart failure	PISP	Health literacy	Patient activation	Not specified	Patient Activation Measure (PAM)-13 questionnaire (40)
			PISP	Health literacy	Self-care maintenance	Not specified	Self-Care of Heart Failure Index (SCHFI): Self-care maintenance scale (41)
			PISP	Health literacy	Self-care management	Not specified	Self-Care of Heart Failure Index (SCHFI): Self-care management scale (41)
			PISP	Health literacy	Self-care confidence	Not specified	Self-Care of Heart Failure Index (SCHFI): Self-care confidence scale (41)
			Medical benefit		Heart failure–related quality of life	Not specified	Minnesota Living With Heart Failure Questionnaire (MLHFQ)(42)
Ahmed et al. 2016 (36)	Randomized controlled trial	Self-management and tele-consultation web intervention for adults with bronchial asthma	PISP	Coping with illness-related difficulties in everyday life	Self-efficacy	2^nd	Chronic Disease Self-Efficacy Scale (43)
			Medical benefit		Asthma control: overuse of fast-acting bronchodilators	1^st	pd: Units recorded in drug database
			Medical benefit		Asthma-related quality of life	1^st	Mini-Asthma Quality of Life Questionnaire (MAQLQ)(44)
			Medical benefit		Depression	2^nd	Patient Health Questionnaire (PHQ-9)(45)
			Medical benefit		Health-related quality of life	2^nd	EuroQol visual analogue scale (EQ-VAS)(46)
			Medical benefit		Severity of asthma symptoms	2^nd	The Asthma Control Test (ACT)(47)
			Other		Acceptability and attitude of participants toward platform	2^nd	Self-developed/adapted (theory-based)
			Other		Beliefs about medication	2^nd	Beliefs about Medicines Questionnaire (BMQ)(48)
			Other		Usage of self-management platform	2^nd	pd: Platform tracking data
Kleinman et al. 2017 (34)	Randomized controlled trial	Self-management and tele-consultation web and mobile intervention for adults with type 2 diabetes mellitus	PISP	Adherence	Frequency of blood glucose testing	2^nd	pd: Share of blood glucose tests according to testing schedule
			PISP	Adherence	Medication adherence	2^nd	pd: Took all medications last week according to personalized medication list
			PISP	Alignment of treatment with guidelines and recognized standards	Compliance with quarterly schedule according to treatment guideline	2^nd	pd: Share of patients taking recommended quarterly schedule
			Medical benefit		Blood glucose level (A1c)	1^st	pd: Blood test
			Medical benefit		Body mass index	2^nd	pd: Physical measurement
			Medical benefit		Fasting blood glucose	2^nd	pd: Blood test
Skrepnik et al. 2017 (35)	Randomized controlled trial	Self-monitoring mobile intervention foradults with knee osteoarthritis	PISP	Health literacy	Patient activation	2^nd	Patient Activation Measure (PAM)-13 questionnaire (40)
			Medical benefit		Mobility	1^st	Steps per day
			Medical benefit		Distance	2^nd	pd: 6-minute walk test: distance
			Medical benefit		Mood state	2^nd	Visual Analog Mood Scale (VAMS)(49)
			Medical benefit		Pain	2^nd	pd: 6-minute walk test: pain
			Medical benefit		Sleep duration	2^nd	pd: Tracking data by wearable
			Other		Patient and physician satisfaction with treatment via device	2^nd	Self-developed/adapted (theory-based)
Kryger et al. 2019 (37)	Randomized controlled trial	Self-management mobile intervention for adults with spinal cord injury	PISP	Coping with illness-related difficulties in everyday life	Self-management and independence	2^nd	Adolescent Self-Management and Independence Scale (50)
			PISP	Reduction of therapy-related efforts and strains for patients and their relatives	Perceived physical independence	2^nd	Physical independence domain of the Craig Handicap Assessment and Reporting Technique Short Form (51)
			Medical benefit		Emergency department visits	1^st	pd: Number of encounters in the emergency department for any reason
			Medical benefit		Emergency department visits because of urinary tract infections or pressure injuries	1^st	pd: Number of encounters in the emergency department specifi-cally for urinary tract infection or pressure injury diagnosis, evaluation, or treatment
			Medical benefit		Hospitalizations	1^st	pd: Number of admissions to the hospital for any reason
			Medical benefit		Hospitalizations because of urinary tract infections or pressure injuries	1^st	pd: Number of admissions to the hospital specifically for urinary tract infection or pressure injury diagnosis, evaluation, or treatment
			Medical benefit		Pressure injuries	1^st	pd: Number of unique episodes of skin breakdown
			Medical benefit		Urinary tract infections	1^st	pd: Number of urinary tract infections with positive urine cultures
			Medical benefit		Activities of daily living	2^nd	Canadian Occupational Performance Measure (COPM)(52)
			Medical benefit		Depression	2^nd	Beck Depression Inventory-II (BDI-II)(53)
			Medical benefit		Health-related quality of life	2^nd	World Health Organization Quality of Life Brief Instrument (54)
			Other		Experience and satisfaction with chronic care	2^nd	Patient Assessment of Chronic Illness Care (PACIC)(55)
Nyberg et al. 2019 (38)	Controlled pragmatic pilot trial	Digital health education web intervention for adults with chronic obstructive pulmonary disease (COPD)	PISP	Coping with illness-related difficulties in everyday life	Exercise self-efficacy	2^nd	Spinal Cord Injury Exercise Self-Efficacy Scale (ESES)(56)
			PISP	Health literacy	Confidence in self-managing COPD	2^nd	Self-developed/adapted (theory-based)
			PISP	Health literacy	Health literacy	2^nd	Swedish Critical Health Literacy (C&CHL) Scale (57)
			Medical benefit		Impact of COPD in daily life	1^st	COPD Assessment Test (CAT)(58)
			Medical benefit		Dyspnea	2^nd	Modified Medical Research Council (mMRC) scale (59)
			Medical benefit		Health-related quality of life	2^nd	EuroQol-5D (EQ-5D)(60)
			Medical benefit		Physical activity	2^nd	Grimby’s Activity Scale (Grimby PA)(61)
			Medical benefit		Physical activity and exercise	2^nd	Swedish National Board of Health and Welfare indicator questions for PA & exercise (SOS-PA)(62)
			Medical benefit		Physical inactivity	2^nd	Swedish National Board of Health and Welfare scale indicator question for physical inactivity (62)

1^st – primary outcome, 2^nd – secondary outcome, pd - process-generated/tracked/directly measured data

Table 4: Overview of validated outcome measurement instruments by PISP outcome domains used in studies on DiGA-compliant applications

PISP outcome domain	Outcome	Validated outcome measurement instrument
Health literacy	Health literacy	Swedish Critical Health Literacy (C&CHL) Scale(57)
	Patient activation	Patient Activation Measure (PAM)-13 questionnaire(40)
	Self-care maintenance	Self-Care of Heart Failure Index (SCHFI): Self-care maintenance scale(41)
	Self-care management	Self-Care of Heart Failure Index (SCHFI): Self-care management scale(41)
	Self-care confidence	Self-Care of Heart Failure Index (SCHFI): Self-care confidence scale(41)
Coping with illness-related difficulties in everyday life	Self-efficacy	Chronic Disease Self-Efficacy Scale(43)
	Exercise self-efficacy	Spinal Cord Injury Exercise Self-Efficacy Scale (ESES)(56)
	Self-management and independence	Adolescent Self-Management and Independence Scale(50)
Reduction of therapy-related efforts and strains for patients and their relatives	Perceived physical independence	Physical independence domain of the Craig Handicap Assessment and Reporting Technique Short Form(51)
Adherence	No validated outcome measurement instruments used in studies on DiGA-compliant applications
Alignment of treatment with guidelines and recognized standards
Coordination of treatment procedures
Facilitating access to care
Patient autonomy
Patient safety

Competing interest reported. MS, AK and LH declare that there is no conflict of interest. JS received institutional grants for investigator-initiated trials from Novartis, Sanofi, ALK, and Pfizer. He acted as a consultant for Novartis, ALK, Lilly, and Sanofi.

AdditionalFile1InandExclusioncriteriaandsearchstringinitialsystematicreview.docx
Additional file 1: Inclusion and exclusion criteria and search string of the intial review
AdditionalFile2Dataextractiontableforallincludedstudies.xlsx
Additional file 2: Data extraction table for all included studies

Outcome domains and measurement instruments of patient- relevant improvement of structure and processes as a new set of outcomes for evaluating and approving digital health applications: systematic review

Status:

Version 1

Abstract

Figures

Background

Definition of DiGA

Evaluation of positive healthcare effects

Research gaps concerning PISP

Methods

Research questions

Search strategy

Eligibility criteria

Study selection and data extraction

Results

Study selection

Study characteristics

Characteristics of outcome core areas, outcome domains, outcomes, and outcome measurement instruments

Discussion

Main findings in the context of previous research

Implications for future research

Implications for practice

Limitations

Conclusions

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors’ contribution

Acknowledgements

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1