Methodologies for the benefit-risk analysis of medical devices: A systematic review

doi:10.21203/rs.3.rs-4832842/v1

Download PDF

Systematic Review

Methodologies for the benefit-risk analysis of medical devices: A systematic review

https://doi.org/10.21203/rs.3.rs-4832842/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Introduction: The use of medical devices (MDs) by patients carries both benefits and risks. Regulatory frameworks such as the EU Medical Device Regulation (MDR) and the US Federal Food, Drug, and Cosmetic Act (FD&C Act) mandate the systematic weighing of these benefits and risks through a benefit-risk analysis (BRA). This systematic review aims to identify existing BRA methodologies for MDs, and evaluate their strengths and weaknesses.

Methods: A literature search was conducted in PubMed, Scopus, Web of Science, Google Scholar, and Semantic Scholar, covering publications from 2000 onwards using a search string that contained the search terms (1) methodologies AND (2) benefit-risk analysis AND (3) medical device OR in vitro diagnostic medical device. Peer-reviewed publications were included when they described BRA methodologies for MDs. The exclusion criteria included records on unrelated technologies, pharmaceuticals, non-English publications, and insufficient descriptions. The quality assessment was performed using a method proposed by Hawker et al. (2002). The methods and their characteristics were narratively summarised. Each method was assessed for its degree of objectivity and for using numerical calculations on a scale from 0 to 5.

Results: The search identified 622 records, with six meeting the inclusion criteria. The included studies described ten BRA methodologies: Multicriteria Decision Analysis (MCDA), Health Outcomes Modelling (HOM), Stated-Choice Surveys (SCS), Composite Net Clinical Outcome (CNCO), Pairwise Comparisons (PC), Complete Profile of Benefits and Risks (CPBR), Net Benefit Score and Benefit-Risk Ratio (NBS & BRR), Quantitative Benefit-Risk Assessment (qBRA), Quantitative Benefit-Risk Determination (QBRD), and the FDA’s Benefit-Risk Framework (BRF). Methods varied in their reliance on numerical calculations and their degree of objectivity, with a significant correlation between both. The average level of objectivity was assessed as medium (2.1/5).

Discussion: This review describes the ongoing debate between qualitative and quantitative BRA methods. While qualitative methods are often criticised for being subjective and biased, our findings reveal that quantitative methods, though more objective, still exhibit subjectivity, especially in endpoint identification and relevance assignment. Despite the rise of quantitative methods, often originating from the pharmaceutical industry, qualitative methods are often still used by MD decision-makers. Additionally, several quantitative methods are unsuited for certain difficult-to-quantify risks of MDs. Integrating both qualitative and quantitative elements may offer a more comprehensive BRA framework for MDs.

Translational Medicine

Benefit-Risk Analysis

Medical Devices

Systematic Review

Methodologies

PRISMA

Regulation

Risk Management

The use of medical devices (MDs) inherently carries risks in addition to the intended benefits for patients. One main objective of quality processes that are used in the development of MDs and their related regulatory approval processes is to identify and assess these risks and then mitigate them in a manner which brings them under acceptable control [1–3]. While some risks can be completely eliminated, others are only reduced in likelihood or severity, referred to as ‘residual risks’[3]. To determine whether a MD is safe and effective and should be made available on the market, many international MD legal frameworks, such as the EU’s Medical Device Regulation (MDR) [1] and the Federal Food, Drug & Cosmetic Act in the US (FD&C Act) [4] require manufacturers and regulators to systematically weigh the remaining risks against the benefits of the MD. This ‘benefit-risk-analysis’ (BRA), also referred to as ‘benefit-risk assessment’ or ‘benefit-risk determination’, is therefore a crucial step in the MD approval process and throughout the MDs entire lifecycle [1–3, 5].

Both the FD&C Act in the US as well as the MDR in the EU state that BRA is required and define high-level principles of how it should be conducted. Neither legislation sets out specific methods [1, 4]. These regulations are defined in more detail in official guidelines or by reference to recognised international standards [1, 4]. For the US, multiple guidance documents issued by the Federal Drug and Cosmetic Agency (FDA) describe the concept, goal, and factors that should be considered within a BRA and define a method known as the FDA’s benefit-risk framework (BRF) [2]. Additionally, those guidance documents also indicate the applicability of ISO 14971:2019, the recognised standard for the risk management of MDs, which describes the BRA as part of the risk management process [3]. In the EU, reference is instead made to the harmonised version EN ISO 14971:2019 [6], which is identical in terms of content. The technical report ISO TR 24971:2020, which describes the implementation of the framework provided in ISO 14971:2019, provides further guidance and examples [7]. However, although these documents provide a general framework for the BRA, they do not specify a detailed method. It is, therefore, likely that different methods, qualitative or quantitative nature, are used in the approval processes of MDs both in the EU and the US.

In recent years, the methods used for BRA in the pharmaceutical sector moved from qualitative and subjective approaches to more structured, quantitative and objective frameworks [8, 9]. This shift is primarily enabled by the fact that the risks and benefits of pharmaceuticals are comparatively easy to quantify, as they usually have a narrow indication with an intended benefit that could serve as a clinical endpoint in a trial, making statistical evaluation more straightforward [10]. This similarly applies to the risks associated with a medication, as adverse events and negative effects are registered in clinical trials and also evaluated. For MDs, however, implementing such quantitative methods is considerably challenging since they often have a broader intended use (an example of this is an ultrasound scanner that can be used for multiple purposes in multiple conditions and on multiple body regions), and require considering stakeholder perspectives, accounting uncertainty, and the crucial role of usability, training and experience to assess benefits and risks accurately [3, 10, 11]. Additionally, innovative MDs often pose risks that were previously unknown and difficult to quantify, e.g., risks associated with cybersecurity [12], gamification of interfaces [13], artificial intelligence (AI) [14, 15], or virtual reality [16, 17]. Current guidance documents acknowledge this difficulty and the complexity of assessing such risks quantitatively [2, 3, 7, 11, 18].

While the relevance of the BRA is acknowledged, and several methods for the conduct of a BRA have been published [5, 10], there is currently no definitive standard that defines how the BRA should be conducted, and that is applicable to developers and regulators [10, 19]. This is paradoxical, as the market approval and surveillance approaches of MDs heavily rely on the concept of BRA [1, 4, 5, 10]. To address the existing ambiguities around the methods on how to conduct a BRA, this systematic review primarily aims to (1) identify existing BRA methodologies for MDs and (2) describe the weaknesses and strengths of those methods.

To answer its primary research questions, this systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) reporting guidelines [20] and was pre-registered on February 22, 2024, with Open Science Framework Registries under the following DOI: 10.17605/OSF.IO/RT4SV. The completed PRISMA checklist can be found in Supplemental Material 1. The protocol has been updated twice. Updates are accessible in the registry. Additional deviations to the protocol are listed in Supplemental Material 1.

Search strategy

The literature search was conducted from February 27, 2024, to April 05, 2024, using the following electronic databases: PubMed, Scopus, WebOfScience, Google Scholar, and Semantic Scholar, covering articles published after the year 2000 due to major changes in the MD regulations in the US and EU in recent years and since the BRA concept is comparatively novel [8]. A standardised search term was developed consisting of three main sub-term categories: (1) methodologies AND (2) Benefit-risk analysis AND (3) MDs or IVDs. A description of the complete search strings, adapted for each database, can be found in Supplemental Material 1. Additionally, the reference lists of included articles were screened to identify additional relevant records. The bibliographic data of all identified publications was exported and gathered in an electronic database. Duplications were removed semi-automatic by a human reviewer comparing the bibliographic metadata of a record.

Eligibility criteria

Records were eligible for inclusion if they met the following inclusion criteria: (1) The record addressed benefits, risks, performance, and/or safety in MDs; (2) The record describes and/or discusses BRA methodologies for MDs, (3) the record was peer-reviewed. Records were excluded: (1) if they were focussed on unrelated technologies or products; (2) when the information provided was insufficient to undertake a scientific analysis; (3) when they were not available in English; (4) when the full text was not available; (5) when they were published before 2000.

Study selection and data extraction

After the removal of duplicates, the titles and abstracts of the identified articles were screened by two independent researchers to evaluate whether they met the criteria for inclusion. In the case of disagreements, a third independent reviewer was consulted. The screening process was performed by using rayyan.ai [21] between April 5th, 2024, and May 31, 2024. After the initial screening, two reviewers screened the full text of the included publications for suitability.

One reviewer extracted the data of all included records using the software MAXQDA 2022 [22]. The extracted data included a description of the method, context, rationale, procedures, type of device or technology, and the methodological strengths and limitations identified by the authors of each record. At this stage, the references cited by each record were screened for relevance, adhering to the same inclusion and exclusion criteria.

Quality assessment

As a risk of bias analysis was not feasible for this type of evidence, the quality assessment method ‘Appraising the Evidence’ for qualitative health research, initially proposed by Hawker et al. [23], was used. The detailed quality assessment checklist for each included record can be found in Supplemental Material 1.

Data synthesis and analysis

For the synthesis and analysis, one reviewer categorised the content of the records into broad methodological themes, considering the extracted data. The methods and their characteristics were then narratively summarised using figures, tables and narrative summaries. Additionally, an assessment of each method was conducted to identify the type and degree of both numerical and subjective components involved. This assessment used a two-dimensional framework, with one axis representing the extent to which a method relies on mathematical calculations (numerical vs. non-numerical) and the other axis evaluating the degree of human judgment involved (subjective vs. objective). Methods that heavily utilised mathematical calculations to determine the relevance and relation of benefits and risks were categorised as highly numerical, while those that did not employ mathematical calculations were classified as non-numerical. Concurrently, methods that applied stringent, automatic criteria for the identification and weight assignment without human judgment were considered highly objective, whereas those that depended solely on human judgment without any formalised system were deemed highly subjective. Each reviewer had to rate the objectivity and use of numerical calculations on a scale from 0 (lowest) to 5 (highest) based on predefined criteria. A complete description of the assessment framework can be found in Supplemental Material 1. The assessment was conducted by three independent reviewers (authors F.J. and M.O.), with a third reviewer as the referee in case of deviations (Author O.F.). The statistical analysis was conducted using Python version 3.11.6 and the following libraries: pandas, and SciPy. Spearman's rank correlation coefficient and significance were calculated to test the correlation between the use of numerical methods and objectivity [24].

Search results

The systematic search of five databases identified 622 records, from which 137 duplicates were removed. Of the remaining 485 records, 460 were excluded based on the screening of the title and abstract, and another 21 were excluded based on the assessment of the full text. In total, four records were included in the analysis through the systematic search process, an additional two were included via citation search. Figure 1 presents the selection process of the records as a PRISMA Flowchart.

Characteristics of included papers and Quality Assessment

The included records describe one or more BRA methods for MDs each. Four documents describe multiple methods [10, 25–27]. A large percentage of the authors come from the US (4/6) [10, 25–27]. All papers were published since 2014, with four of them published after 2020. The quality of the included records was mostly rated as ‘Fair’, with one record being ranked as ‘Good’. Most of the included records had weaknesses in their sampling strategy and transferability. Some records describe methods but only provide vague definitions of them [10, 25]. This is, to some degree, reflected in the quality assessment of the paper describing the methods. Table 1 summarises the key characteristics of these studies, including their authors, year of publication, country, described methods, and quality assessment.

Table 1

Characteristics of included reports and studies, ordered by the year of publication.
Study	Year	Origin	Described Methods	Quality Rating
Agapova et al. [10]	2014	USA	Multicriteria Decision Analysis (MCDA); Health outcomes modelling (HOM); Stated-choice surveys (SCS)	Fair
Colopy et al. [25]	2015	USA	Composite net clinical outcome (CNCO); Pairwise Comparisons (PC); Complete profile of benefits and risks (CPBR)	Fair
Sun et al. [26]	2020	USA	Net Benefit Score and Benefit-Risk Ratio (NBS & BRR);	Good
Fu et al. [27]	2020	USA	Quantitative Benefit-Risk Assessment (qBRA), MCDA	Fair
Su & Deng [5]	2023	China	MCDA; NBS & BRR; Quantitative Benefit-Risk Determination (QBRD); FDA’s Benefit–Risk Framework (BRF)	Fair
Tervonen et al. [28]	2023	Switzerland	MCDA	Fair

BRA Methods for Medical Devices

The six documents included in our study described ten individual BRA methods for MDs: Multicriteria Decision Analysis (MCDA) [5, 10, 27, 28], Health Outcomes Modelling (HOM) [10], Stated-Choice Surveys (SCS) [10], Composite Net Clinical Outcome (CNCO) [25], Pairwise Comparisons (PC) [25], Complete Profile of Benefits and Risks (CPBR) [25], Net Benefit Score and Benefit-Risk Ratio (NBS & BRR) [5, 26], Quantitative Benefit-Risk Assessment (qBRA) [27], Quantitative Benefit-Risk Determination (QBRD) [5], and the FDA’s Benefit-Risk Framework (BRF) [5]. MDCA was discussed in four documents [5, 10, 27, 28], while NBS & BRR was discussed in two documents [5, 26]. Eight methods (MCDA, NBS & BRR, HOM, SCS, CNCO, CBPR, qBRA, PC) originate from the pharma sector and were initially designed to assess the benefits and risks of pharmaceuticals but could extended to MDs [5, 10, 25–28]. BRF was initially proposed by the FDA and is used for their regulatory decisions [2, 5]. NBS & BRR is based on a similar FDA framework for drugs and biological products [29] and describes itself as adding quantitative aspects to it [26]. Three methods are further developments of existing ones, which either add quantifiable elements (NBS & BRR, qBRA) [26, 27] or allow an evaluation of MDs in the first place (MCDA) [5].

MCDA is a structured and transparent method, allowing for the comparison of multiple alternatives by evaluating conflicting criteria [5, 10, 28]. However, it is complex to implement, relies on subjective weight assignments, and requires comprehensive data sources [5, 10]. NBS & BRR provides a calculated score for a product’s benefit-risk profile but has a potential for oversimplification of complex benefits and assumes equal weights for all benefits and risks [5, 26]. QBRD is a highly numerical method that calculates the benefit-risk profile based on pairwise comparisons [5, 30]. Despite reducing subjectivity, it requires comprehensive data collection and struggles with non-quantifiable data [5]. HOM aggregates multiple benefits and risks into a single metric, such as quality-adjusted life years (QALY), which helps in understanding the long-term effects of a drug or device. Its limitations include the requirement for extensive quantitative data, the complexity of computations, and less transparency [10]. SCS assesses stakeholder perspectives on the benefit-risk profile of a product based on choice experiments but faces challenges with sample size requirements and potential discrepancies between experimental choices and real-life decisions [10]. CNCO combines multiple endpoints into a single metric but assumes clinical equivalence of endpoints and does not consider multiple risks for one patient [25]. Similar to QBRD, PC compares a single benefit with a single risk endpoint [5, 25]. However, it is not as structured as QBRD, is limited when multiple endpoints are involved, and provides no clear judgment of the benefit-risk profile [25]. CPBR provides a transparent overview of all relevant endpoints by comparing them between a test device and a control group. However, it can be complex due to the high number of endpoints and assumes equivalence of all endpoints [25]. qBRA uses functions and statistical analyses for a structured and transparent assessment but requires subjective identification of relevant endpoints and involves complicated calculations [27]. BRF, provided and used by the FDA, is a qualitative method with a systematic approach for evaluating MDs benefits and risks BRF [2, 5], but it could be subjective and untransparent [5, 26]. Table 2 provides an overview of the included methods, their strength and limitations.

Table 2

**Methods for the benefit-risk analysis of medical devices.** This table provides an overview of the methods identified through the systematic search. For each method, a summary and description, and their limitations and strengths, as described by the authors, are provided.
Name	Summary	Description	Limitations	Strengths
MCDA [5, 10, 26, 28]	MCDA is a structured framework that can calculate and compare BRA scores of multiple alternatives.	For each alternative, benefit and risk criteria are defined. Each criterion is given a score based on performance data and a predefined performance matrix. The performance scores are multiplied with a relative weight for the criteria, predefined based on stakeholder perspectives. The weighted scores are summed. This total score can be compared with the total score of the alternatives.	• Criteria are treated as independent • Complex to implement due to diverse data sources • Subjectivity in assigning weights • Subjectivity in defining benefit and risk criteria • Data from post-market surveillance and literature may vary in quality and reliability • Requires a control group/alternatives for comparison	• Structured presentation • Transparency in assessing weights and in presenting clinical-related benefit and risk indicators • Supports subsequent sensitivity analysis to test the weight dependence • A single score for each alternative allows comparison • Flexible and adaptable to various scenarios • Considers uncertainty
NBS & BRR [5, 26]	NBS & BRR is a quantitative version of the FDA’s BRF. It could be used to calculate a Net Benefit Score (NBS) and a Benefit-Risk Ratio (BRR) for a MD. This score shows whether the benefits outweigh the risks.	All benefits, risks, and risk mitigations are rated with a score based on the severity and prevalence of the condition, the availability of alternatives, and the robustness of the data. The NBS is calculated by the sum of the benefit scores minus the sum of the risk scores (taking the mitigation of each risk into account). The BRR is the sum of the benefit scores divided by the sum of the risk scores (taking the mitigation of each risk into account).	• May oversimplify complex benefits. • May not capture all risk factors. • Assumes equal weight for all benefits and risks.	• Provides a numerical value for the benefits, risks and how they relate to each other. • Considers risk mitigation. • Allows to predict FDA decisions.
QBRD [5, 30]	QBRD identifies specific benefits and risks, assigns values, calculates benefit-risk ratios, and compares them to the state of the art.	Quantifiable risks and benefits are identified and get a numerical value assigned (magnitude for benefits, severity for risks) based on a predefined matrix. The benefit and risk values are then calculated by multiplying the numerical value with the frequency of occurrence. Each benefit will be pairwise compared with each risk by calculating a pair-specific benefit-risk ratio. This benefit-risk ration will be compared with the state-of-the-art and with pre-defined acceptance criteria.	• Requires comprehensive data collection. • Subjectivity in determining magnitude values. • Complex for devices with multiple benefits and risks. • Does not integrate all benefits and risks.	• Reduces subjectivity • Ensures clarity and specificity in the analysis. • Allows a quantitative comparison. • Provides a clear metric for decision-making.
HOM [10]	HOM uses a disease modelling approach where the benefit-risk profile is determined based on a single metric that describes the impact of a device on this disease.	A measure that is used for disease modelling must be chosen, e.g., quality-adjusted life years (QALY). This measure is then adjusted by the benefits and risks (e.g., QALYs gained by the benefits and lost through the risks) based on quantitative and qualitative data. The impact of the device could then be expressed through the net health benefit.	• Data may be limited for specific health states. • Requires extensive quantitative data. • Computationally intensive. • Utility weights may not always be available. • Untransparent results due to the combination of multiple benefits and risks into one metric. • Regulators were not very receptive to epidemiological methods such as QALYs.	• Can incorporate multiple benefits and risks. • Aggregates outcomes over long periods. • Incorporates uncertainty in parameter values. • Helps decision-makers understand long-term outcomes.
SCS [10]	This method quantifies stakeholder preferences by presenting choices between different attributes of treatments.	After the test device is defined, a number of choices, which represent the risk profile in abstract form, is created. Multiple stakeholders are presented with this choice experiment. From the survey data, weights and preferences for the benefit-risk profile could be calculated using statistical methods	• Respondents may struggle with the understanding of rare risks. • Sample size requirements can be large. • Combining the results across multiple criteria is complex and requires advanced statistical techniques. • Subjects could decide differently in the experiment than they would in real life.	• Captures preferences for specific attributes. • Quantifies stakeholder preferences for benefits and risks. • Estimates willingness to accept risks for benefits. • Provides clear metrics for decision-making. • Considers multiple stakeholder perspectives. • Considers uncertainty.
CNCO [25]	This method combines multiple benefit-risk endpoints into a single metric.	Relevant endpoints are identified and combined into a single endpoint. This endpoint could then be measured, e.g., with the time-to-first event method.	• Endpoints are considered as clinically equivalent. • Does not consider multiple risks for one patient.	• Is relatively simple to calculate.
PC [25]	This method compares a single endpoint related to a benefit to a single endpoint related to a risk.	A benefit and a risk related endpoint are chosen. A numerical value is assigned to each endpoint, e.g., based on a clinical trial or stakeholder interviews. Both values could be compared with each other to assess whether one outweighs the other.	• Limited when trade-offs involve multiple endpoints. • Treats separate events as clinically equivalent.	• Provides clear values for a benefit and risk pair • Could use different comparison metrics. • Could be used for several benefit and risk types. • Easy to understand and communicate.
CPBR [25]	This method provides an overview of all benefit- and risk-related endpoints.	In this method, the benefit-risk profile of a test device is compared with a control group. All relevant benefit- and risk-related endpoints are defined. The probability of the occurrence of those endpoints is compared between the two groups. The risk difference between both groups could be shown in a forest plot.	• Can be complex to assess due to the high number of endpoints. • Needs a comparison/control group. • Deciding to favour one device could be difficult if some endpoints favouring one device, and the rest the control. • Considers all endpoints as equivalent as a default.	• Provides a transparent overview of all endpoints. • Allows an individualised assessment.
qBRA [27]	This method calculates a device’s benefit-risk profile through complex functions and statistical analyses.	Favourable (benefits) and unfavourable (risks) endpoints are defined by a multi-functional team. A standardised score (SC) is calculated with a value function for each endpoint. Based on stakeholder preferences, the SCs are altered with weights. The overall benefit-risk value is then calculated with a utility function. To obtain numerical results, stochastic methods can be applied.	• Subjective identification of relevant endpoints. • Needs comparison/control groups. • Involves complex calculations	• Is structured and transparent. • Considers uncertainty. • Could be used for multiple use cases by choosing value and utility functions.
BRF [2, 5]	This qualitative method is used by the FDA to determine whether the benefits of a device outweigh its risks.	The BRF provides a worksheet and examples that could be followed. Initially, the type, magnitude, probability, and duration of the benefits are listed, and the uncertainty is assessed. Then, the risks and their severity, probability and duration are assessed. Based on this, it is determined, whether the benefits outweigh the risks?	• Is qualitative and subjective. • Could be hard to predict and untransparent.	• Provides a structured and systematic approach for FDA staff.

Assessment of Quantitative and Qualitative Methods

A majority of 70% (7/10) of the methods are described by their authors as quantitative (NBS & BRR, QBRD, HOM, SCS, CNCO, PC, and qBRA) [5, 10, 25–27], 20% (2/10) of described methods are described as both qualitative and quantitative (MCDA and CPBR) [5, 10, 25, 27, 28], and 10% (1/10) of the methods are described as qualitative only (BRF) [5]. While all included papers use this classification, only one provides a clear definition, as follows: qualitative BRA methods are defined as descriptive approaches that focus on individual benefits and risks, including their frequency and duration, and describe how this information influences the decision of whether the benefits outweigh the risks, while quantitative BRA methods integrate endpoints by quantifying their clinical relevance and trade-offs, which usually requires judgment in assigning numerical weights to benefits and risks [25]. This definition was used within the analysis of this review.

Qualitative methods are described as inferior in effectively assessing trade-offs between different outcomes [25, 28], more subjective [5, 26], untransparent [26], and inferior in providing clear results [27, 28]. In contrast, quantitative methods are described as more transparent, rational, and consistent for decision-making [5, 10, 26, 27] by quantifying clinical relevance and trade-offs [5, 25] but can be mathematically complex and potentially subjective in weight assignments [25].

We conducted an assessment to determine whether the methods described used numerical calculations and the degree of objectivity, regardless of how they were categorised by their authors. The use of numerical methods and calculations to determine a device’s benefit-risk profile ranged from not numerical (0) to completely numerical (5). At the same time, the objectivity for identifying benefits, risks, and their relevance ranged from not objective and thus, subjective (0) to nearly objective (4), with a mean value of 2.1 for all methods, and 2.2 for methods classified as quantitative. Spearman's rank correlation coefficient was 0.70 with a p-value of < 0.05, which is interpreted as a strong and significant positive correlation. An outlier here is the SCS, which relies heavily on numerical calculations based on statistical analyses but makes these on the basis of highly subjective stakeholder statements [10]. Table 3 provides an overview of the assessment.

Table 3

**Assessment Results.** This table shows the results of the assessment of whether the methods described used numerical calculations and the degree of objectivity.
Name	Self-Classification	Use of numerical calculations (0–5)	Objectivity (0–5)
MCDA [5, 10, 26, 28]	Qualitative/ Quantitative	4	3
NBS & BRR [5, 26]	Quantitative	5	3
QBRD [5, 30]	Quantitative	5	4
HOM [10]	Quantitative	3	2
SCS [10]	Quantitative	4	0
CNCO [25]	Quantitative	3	2
PC [25]	Quantitative	3	2
CPBR [25]	Qualitative/ Quantitative	3	1
qBRA [27]	Quantitative	5	3
BRF [2, 5]	Qualitative	0	1

Principal Results

The first objective of this systematic review was to identify existing BRA methods for medical devices described in the scientific literature. We analysed six studies, revealing ten individual BRA methods for MDs. A majority (7) of those methods are described as quantitative, two as both qualitative and quantitative, and one as qualitative only. The quality of the included papers was mixed, ranging from x to y. Several papers describe methods which, however, are only insufficiently defined within those papers (HOM, CNCO, PC, CPBR) [10, 25]. Our assessment of the methods shows that, even when being described as quantitative, the use of mathematical calculations to determine the variables within the BRA ranges from not numerical (0) to completely numerical (5). At the same time, the methods range from nearly subjective (1) to nearly objective (4). We identified a strong positive and significant correlation between objectivity and the use of numerical calculations.

The second objective was to describe the weaknesses and strengths of the included methods as reported in the literature. For example, the MCDA method was noted for its structured approach to evaluating multiple conflicting criteria but was criticised for its reliance on point estimates and the complexity of implementation [5, 10, 27, 28]. The NBS & BRR method provided clear numerical values for benefits and risks but was criticised for potentially oversimplifying complex benefits and risks [5, 26]. Other methods, like HOM, SCS, and qBRA, offered comprehensive approaches but required extensive data and sophisticated analysis techniques [10, 27].

Comparing Qualitative and Quantitative BRA Methods

Multiple authors criticise the qualitative nature of some current methods, such as the BRF, as being subjective, susceptible to bias and therefore inferior to quantitative methods [5, 26–28]. Our findings partially support this critique since the use of numeric methods correlated with the objectiveness of a method, although there are some outliers. Due to this assumption, the majority (70%) of the methods identified in this systematic review were developed to be quantitative in nature. However, other studies indicate that the most common BRA methods used by MD decision-makers in real-world assessments are qualitative [19, 31–33]. One potential reason for this preference could be that many of the quantitative methods originate from the pharmaceutical industry, where such approaches are well-established and useful [5, 8]. Multiple studies are attempts to adapt these methods for MDs [5, 26, 27], yet some may not be a perfect fit due to fundamental differences between pharmaceuticals and MDs described above [3, 5, 10, 11, 27]. Especially purely quantitative methods, such as PC and QBRD, could only process quantitative data, making them unsuitable for hard-to-quantifiable risks, such as those related to new technologies [12–17]. This limitation is acknowledged in the most recent version of applicable standards [3, 7].

In our assessment of the use of numerical calculations and objectivity within each method, we observed that the extent to which numerical methods were used varied from three to five in the subgroup of quantitative methods. This categorisation as ‘quantitative’ was derived from the self-classification of the authors of the individual methods. However, the strong correlation between numerical methods and the objectivity shown in our analysis and expected by other authors [5, 26–28] does not apply universally; there are methods that, despite being highly numerical, remain very subjective (SCS [10]). Additionally, our assessment of the overall level of objectivity was medium-low, with a mean value of 2.2 for methods labelled as quantitative. The finding, that even methods described as highly quantitative could, in fact, be subjective in their practical application, is supported by other researchers [25]. This could be due to the fact that most methods for the BRA of MDs include some degree of subjectiveness, at least in the identification of endpoints and the assignment of relevance, making complete objectivity potentially impossible.

Practical Implications and Current Gaps

To our knowledge, this is the first review that refers exclusively to BRA methods for MDs. Previous reviews have primarily focused on pharmaceuticals, resulting in a higher number of methods overall, including a greater variety of qualitative BRA methods [9]. However, due to their unique characteristics, not all of those methods could be applied to MDs [3, 5, 10, 11, 27]. Additionally, the BRA of pharmaceuticals relies on the comparison of effects between different groups, e.g., intervention against placebo [9, 25], which is not always possible for MD testing scenarios. In this case, using the current state of the art (SOTA) as a base for comparison is recommended by methods and guidelines [1, 5, 30].

The importance of conducting a BRA is widely recognised and mandated by regulatory requirements and standards [1–4, 7, 34]. In the EU, there is currently no standardised approach or guideline for BRA, creating a considerable gap [5, 10, 19]. In contrast, in the US, the FDA has issued comprehensive guidance documents that describe the qualitative BRF and provide practical examples but leaves open whether quantitative methods may also be used [2]. However, both US and EU regulations recognise the BRA is an integral step for the approval process, in which all risks of a device are weighted against the clinical benefits to determine whether is should be made available on the market [1, 2, 4]. Thus, in standards and guidance, two levels of the BRA exist: first, an individual BRA for each unacceptable risk, and second, an overall BRA [1–3, 7, 11, 18, 35].

Some BRA methods, such as PC and QBRD, rely on pairwise comparisons of benefits and risks, which have some benefits (granularity, transparency) [5, 25, 30] but fall short in several critical aspects. Firstly, MD benefits and risks are not necessarily linked in pairs, making such comparisons impractical in many cases. Secondly, these methods rely on quantitative data, which could be difficult to collect for some risks. Thirdly, conducting the overall BRA required by standards and guidelines [1–3, 7, 11, 18], could be difficult. Lastly, as the number of risks and benefits increases, the complexity of pairwise comparison methods like QBRD becomes challenging.

With new categories of digital and connected MDs, new challenges that might be difficult to assess by some of the methods described in the review arise. It is already recognised that finding numerical data about risks connected to, for example, cybersecurity, user interfaces, or MDs integrated into IT systems could be hard to find [3, 7] and that innovative MDs often pose risks that were previously unknown and are difficult to quantify [12–17]. This is a gap that current BRA methods described in the literature do not address adequately.

While addressing this gap might be difficult, multiple authors state that there is no “one size fits all solution” [9] which covers all types of MDs, all stages of the product life cycle, and all cases in which a BRA might be necessary [9, 10, 19, 25]. Therefore, different methods could be used depending on the level and type of risk. One approach would be to conduct a qualitative BRA first and then assess whether a quantitative BRA should be conducted [28]. Alternatively, combining qualitative and quantitative approaches into one method could be effective. This can be achieved with methods such as MCDA [5, 28] or the BRF [2, 5]. The benefit of such a combined approach is that these frameworks provide a holistic overview of MDs and are capable of incorporating new risks, making it a comprehensive tool for evaluating the safety and efficacy of especially innovative MDs. A benefit of the BRF, in particular, is that it can incorporate quantitative aspects through improvements proposed by various authors [26], thus overcoming the limitations of qualitative approaches (subjectivity and lack of transparency) by introducing consistency and objectivity to benefit evaluations [5, 26].

We suggest that future research should empirically evaluate the methods identified in this review through applying them to a range of example devices (either real devices or realistic and detailed models of devices), as has already been done in part for pharmaceuticals [36]. This would enable the direct analysis and comparison of the strengths and weaknesses of the different methods and would furthermore allow the assessment of which methods should be excluded from application, which should be retained as reliable. It would also identify how the methods could be further refined and developed to be better fit for purpose.

Limitations

This systematic review is subject to several limitations. First, our findings may be influenced by a form of publication bias, where MD manufacturers do not disclose their used BRA method to the public. Second, we included only reports written in English. Although the majority of scientific research is published in English, this may have resulted in the exclusion of pertinent studies in other languages. Third, we limited our review to studies published after 2000. This restriction was necessary due to the major updates to regulations in recent years, but it may exclude valuable historical insights. Fourth, there is a scarcity of methods specifically designed for medical devices, with most methodologies being developed for pharmaceuticals. The significant differences between these fields mean that many pharmaceutical methods may not be applicable to MDs. Fifth, our review focused exclusively on the EU and the US. This geographic limitation may reduce the applicability of our findings to other regions. Sixth, while research indicates that companies often employ qualitative approaches [32], these are infrequently detailed in the scientific literature, potentially limiting our understanding of their use in practice. Seventh, the lack of standardisation of the terminology in the field may have affected the comprehensiveness of our search terms, possibly leading to the omission of relevant reports. These limitations suggest that our findings should be interpreted with caution and highlight the need for further research to gain a complete understanding.

Despite the critical role of the BRA in the regulatory processes of MD approval, there is currently no generally recognised method or standard. Multiple methods have been proposed in the literature, ranging from qualitative to quantitative. While traditionally, mostly qualitative methods have been used for the BRA, the number of quantitative methods, most of which originate from the pharmaceutical sector, has increased. Although the use of these methods correlates with increased objectivity, this was not particularly high overall for the methods analysed. Additionally, the specific aspects of MDs in terms of their intended use and particular risks that are not quantifiable make it difficult to use exclusively quantitative methods. Future approaches may benefit from integrating qualitative and quantitative elements to enhance comprehensiveness and objectivity. Addressing the gaps in current methodologies is essential for advancing the safety and efficacy evaluation of medical devices.

Data availability

Data collected and used in this review is available within the paper and its supplementary information files.

Acknowledgements

Funding

This work was supported by the European Commission under the Horizon Europe Program, as part of the project CYMEDSEC (101094218). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union, nor the granting authorities, can be held responsible for them. Responsibility for the information and views expressed therein lies entirely with the authors.

Acknowledgement of AI Use

During the preparation of this work the author(s) used DeepL, Grammarly, and ChatGPT (in versions GPT-3.5, GPT-4, GPT-4o) in order to improve grammar, spelling, and readability of the manuscript. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.

Author information

Contributions

O.F. and F.J.contributed equally to this paper. O.F., S.G., and F.J. developed the design of the study. F.J. created the protocol. F.J. conducted the search. F.J., M.O., and O.F. conducted the screening. O.F. and F.J., conducted the data extraction. F.J., M.O. and O.F., conducted the analysis. F.J. and O.F. wrote the first draft of the manuscript. O.F., and F.J. conducted the quality assessment. F.J., M.O., and O.F., conducted the assessment of the methods. F.J., O.F., M.O., M.F., and S.G. contributed to the writing, interpretation of the content, and editing of the manuscript, revising it critically for important intellectual content. All authors had final approval of the completed version and take accountability for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to : Oscar Freyer, [email protected], TUD Dresden University of Technology, Else Kröner Fresenius Center for Digital Health, Fetscherstr. 74, 01307 Dresden

Competing interests

S.G. is an Advisory Group member of the EY-coordinated ‘Study on Regulatory Governance and Innovation in the field of Medical Devices’ conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut für Management GmbH, High-Tech Gründerfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests.

Parliament E, European Council (2017). Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) 178/2002 and Regulation (EC) 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance)Text with EEA relevance
U.S. Food and Drug Administration (FDA) (2019) Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications
International Organization for Standardization (2019) ISO 14971:2019
U.S (1938) Congress. United States Code: Federal Food, Drug, and Cosmetic Act
Su G, Deng D (2023) Regulatory requirements and optimization of multiple criteria decision analysis to quantify the benefit-risk assessment of medical devices. Expert Rev Med Devices 20:273–281. 10.1080/17434440.2023.2190021
Deutsches Institut für Normung DIN EN ISO 14971:2022-04
International Organization for Standardization (2020) ISO/TR 24971:2020
Juhaeri J (2019) Benefit–risk evaluation: the past, present and future. Ther Adv Drug Saf 10:2042098619871180. 10.1177/2042098619871180
Mt-Isa S, Hallgreen CE, Wang N et al (2014) Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 23:667–678. 10.1002/pds.3636
Agapova M, Devine EB, Bresnahan BW et al (2014) Applying Quantitative Benefit–Risk Analysis to Aid Regulatory Decision Making in Diagnostic Imaging: Methods, Challenges, and Opportunities. Acad Radiol 21:1138–1143. 10.1016/j.acra.2014.05.006
Food US, Drug Administration (FDA) (2019) Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian. Device Exemptions - Guidance for Industry and Food and Drug Administration Staff
Perakslis E, Ginsburg GS (2021) Digital Health—The Need to Assess Benefits, Risks, and Value. JAMA 325:127–128. 10.1001/jama.2020.22919
Gilbert S, Freyer O, Wondra K (2024) The regulatory status of health apps that employ serious games and gamification
Freyer O, Wiest I, Kather JN et al A future role for large language model health applications depends on regulators being brave enough to enforce safety standards (In Press). Lancet Digit Health
Meskó B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. Npj Digit Med 6:1–6. 10.1038/s41746-023-00873-0
Freyer O, Gilbert S (2023) Bridging between hype and implementation in medical extended reality. Npj Digit Med 6:1–3. 10.1038/s41746-023-00972-y
Souchet AD, Lourdeaux D, Pagani A et al (2023) A narrative review of immersive virtual reality’s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Real 27:19–50. 10.1007/s10055-022-00672-0
U.S. Food and Drug Administration (FDA) (2016) Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions -. Guidance for Industry and Food and Drug Administration Staff
Smith MY, van Til J, DiSantostefano RL et al (2021) Quantitative Benefit–Risk Assessment: State of the Practice Within Industry. Ther Innov Regul Sci 55:415–425. 10.1007/s43441-020-00230-3
Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. 10.1136/bmj.n71
Rayyan (2021) Rayyan – Intelligent Systematic Review - Rayyan
MAXQDA (2022) MAXQDA | All-In-One Qualitative & Mixed Methods Data Analysis Tool
Hawker S, Payne S, Kerr C et al (2002) Appraising the Evidence: Reviewing Disparate Data Systematically. Qual Health Res 12:1284–1299. 10.1177/1049732302238251
Spearman C (1904) The Proof and Measurement of Association between Two Things. Am J Psychol 15:72–101. 10.2307/1412159
Colopy MW, Damaraju CV, He W et al (2015) Benefit-Risk Evaluation and Decision Making: Some Practical Insights. Ther Innov Regul Sci 49:425–433. 10.1177/2168479014565469
Sun S, Heske S, Mercadel M et al (2021) Predicting Regulatory Product Approvals Using a Proposed Quantitative Version of FDA’s Benefit–Risk Framework to Calculate Net-Benefit Score and Benefit–Risk Ratio. Ther Innov Regul Sci 55:129–137. 10.1007/s43441-020-00197-1
Fu B, Li X, Scott J et al (2020) A new framework to address challenges in quantitative benefit-risk assessment for medical products. Contemp Clin Trials 95:106073. 10.1016/j.cct.2020.106073
Tervonen T, Veldwijk J, Payne K et al (2023) Quantitative Benefit-Risk Assessment in Medical Product Decision Making: A Good Practices Report of an ISPOR Task Force. Value Health 26:449–460. 10.1016/j.jval.2022.12.006
U.S. Food and Drug Administration (FDA). Benefit-Risk Assessment for New Drug and Biological Products. (2023)
Chung B, Kutty J (2022) Benefit-Risk Determination: A Quantitative Approach
Kheir O, Jacoby A, Verwulgen S (2022) Risk Identification and Analysis in the Development of Medical Devices Among Start-Ups: Towards a Broader Risk Management Framework. Med Devices Auckl NZ 15:349–363. 10.2147/MDER.S375977
Gebel M, Renz C, Rodriguez L et al (2024) A Survey to Assess the Current Status of Structured Benefit-Risk Assessment in the Global Drug and Medical Device Industry. Ther Innov Regul Sci 58:756–765. 10.1007/s43441-024-00650-5
Leong J, McAuslane N, Walker S et al (2013) Is there a need for a universal benefit-risk assessment framework for medicines? Regulatory and industry perspectives. Pharmacoepidemiol Drug Saf 22:1004–1012. 10.1002/pds.3464
Medical Device Coordination Group (MDCG) (2019) MDCG 2019-16 Guidance on Cybersecurity for medical devices
International Electrotechnical Commission (IEC) (2021) IEC 80001-1:2021
Hughes D, Waddingham E, Mt-Isa S et al (2016) Recommendations for benefit–risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf 25:251–262. 10.1002/pds.3958

The authors declare potential competing interests as follows: S.G. is an Advisory Group member of the EY-coordinated ‘Study on Regulatory Governance and Innovation in the field of Medical Devices’ conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut für Management GmbH, High-Tech Gründerfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests.

SupplementalMaterial.pdf
Supplemental Material

Download PDF

Version 1

posted

You are reading this latest preprint version

Methodologies for the benefit-risk analysis of medical devices: A systematic review

Status:

Version 1

Abstract

Figures

Introduction

Methods

Search strategy

Eligibility criteria

Study selection and data extraction

Quality assessment

Data synthesis and analysis

Results

Search results

Characteristics of included papers and Quality Assessment

BRA Methods for Medical Devices

Assessment of Quantitative and Qualitative Methods

Discussion

Principal Results

Comparing Qualitative and Quantitative BRA Methods

Practical Implications and Current Gaps

Limitations

Conclusion

Declarations

Data availability

Acknowledgements

Funding

Acknowledgement of AI Use

Author information

Contributions

Corresponding author

Competing interests

References

Additional Declarations

Supplementary Files

Status:

Version 1