ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): A Screening Tool to Assess the Potential for Real-World Data Sources to Support Creation of Credible Real-World Evidence for Regulatory Decision-Making

doi:10.21203/rs.3.rs-3025202/v1

Download PDF

Research Article

ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): A Screening Tool to Assess the Potential for Real-World Data Sources to Support Creation of Credible Real-World Evidence for Regulatory Decision-Making

https://doi.org/10.21203/rs.3.rs-3025202/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 29 Nov, 2023

Read the published version in Health Services and Outcomes Research Methodology →

You are reading this latest preprint version

Background: Adoption and use of RWD for decision-making has been complicated by concerns regarding whether RWD was fit-for-purpose or was of sufficient validity to support the creation of credible RWE. This has greater urgency as regulatory agencies begin to use real world evidence (RWE) to inform decisions about treatment effectiveness.

Methods: We developed a practical screening tool to assess the quality of RWD sources using the framework of Modern Validity Theory. While there has been some convergence of conceptual frameworks, consensus has yet to emerge regarding how to specifically evaluate whether RWD is reliable and fit-for-purpose. We developed a screening tool consistent with the current frameworks and how researchers generally evaluate existing RWD sources for research that they intend to submit to regulatory agencies.

Results: The tool has five dimensions: authenticity, transparency, relevance, accuracy, and reliability. Based upon these dimensions, we developed what more detailed information should be sought by researchers to screen potential RWD sources.

Conclusions: Using a hypothetical example of a medical claims data source, we showed that responding to the tool would not require an extraordinary burden or a lengthy document. This RWD screening tool, which is ready for immediate use, is consistent with current conceptual frameworks to assess whether RWD is fit-for-purpose and adds the additional consideration of experience with RWD sources consistent with Modern Validity Theory.

Real-World Data

Real-World Evidence

Data Quality

Fit-for-Purpose

The use of real-world data (RWD) and real-world evidence (RWE) derived from it has been widely adopted by pharmaceutical developers and a variety of decision makers including providers, payers, health technology authorities and regulatory agencies (Berger et al. 2015; Berger et al. 2016; Berger and Crown 2022; Daniel et al. 2018; Zou et al. 2021). Credible RWE can be created from good quality RWD when investigated within well-designed and well-executed research studies (Berger and Crown 2022). Adoption and use of RWD has been complicated by concerns regarding whether particular sources of RWD are of “good quality” and “fit-for-purpose”. These concerns have become more urgent as regulatory agencies are increasingly using RWD as external comparators for randomized clinical trials and are exploring whether non-interventional RWD studies can provide substantial supplementary evidence of treatment effectiveness.

The US Food and Drug Administration (FDA) draft guidance “Assessing Electronic Health Record and Medical Claims Data to Support Regulatory Decision Making” stated that for all study designs, it is important to ensure the reliability and relevance of data used to help support a regulatory decision (FDA 2021a). Reliability included data accuracy, completeness, provenance, and traceability; relevance includes key data elements (exposures, outcomes, covariates) and a sufficient number of representative patients for the study.

The European Medicines Agencies (EMA) has issued a draft framework for data quality for EU Medicines Regulation (EMA 2022). It defines data quality as fitness for purpose to user needs in relation to health research, policy making, and regulation and the data reflect the reality which they aim to represent (TEHDS EU 2022). It divides the determinants of data quality into foundational, intrinsic, and question specific categories. Foundational determinants are those that pertain to the processes and systems through which data are generated, collected and made available. Intrinsic determinants pertain to aspects that are inherent to a specific dataset. Question specific determinants pertain to aspects of data quality that cannot be defined independent of a specific question. It also distinguishes three levels of granularity of data quality: value level, column level, and dataset level. The dimensions and metrics of data quality are divided into the following categories: reliability, extensiveness, coherence, timeliness, and relevance.

Reliability (precision, accuracy, plausibility) evaluates the degree to which the data correspond to reality.
Extensiveness (completeness and coverage) evaluates whether the data are sufficient for a particular study.
Coherence examines the extent to which different parts of a dataset are consistent in the representation and meaning. This dimension is subdivided into format coherence, structural coherence, semantic coherence, uniqueness, conformance, and validity.
Timeliness is defined as the availability of data at the right time for regulatory decision making.
Relevance is defined as the extent to which a dataset presents the elements required to answer a research question.

TransCelerate has issued a simpler framework entitled “Real-World Data Audit Considerations” that is divided into pillars of relevance, accrual, provenance, completeness, and accuracy (TransCelerate 2022). These frameworks are part of an ongoing dialogue among stakeholders from which international standards for “regulatory-grade RWD” will eventually emerge.

In the meantime, there is an immediate need for researchers with varying levels of RWD experience to have a screening tool to help them assess whether potential RWD sources are fit-for-purpose when designing studies whose purpose is to answer questions from regulatory agencies or to support claims regarding benefits and risks of therapies. To this end, we developed such a tool, consistent with above frameworks, that conforms to how researchers generally approach this critical issue and also incorporated concepts from Modern Validity Theory.

We took our cue on the definition of “fit-for-purpose” from the FDA draft guidance on selecting, developing, or modifying fit-for-purpose clinical outcome assessments (COAs) for patient-focused drug development guidance (to help sponsors use high quality measures of patients’ health in medical product development programs) which states that fit-for-purpose in the regulatory context means the same thing as valid within modern validity theory, i.e., validity is “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests,” and that a clinical outcomes assessment is considered fit-for-purpose when “the level of validation associated with a medical product development tool is sufficient to support its context of use”(FDA 2022). The term validity has been defined in epidemiology to be comprised of internal and external validity relating to study design and execution. We designed the RWD screening tool to focus on evaluation of the real-world data itself within the larger framework of modern validity theory (Royal 2017).

After all, as Wilkinson notes, “good data management is not a goal in itself, but rather is the key conduit leading to knowledge discovery and innovation, and to subsequent data and knowledge integration and reuse by the community after the data publication process” (Wilkinson 2016). Wilkinson proposed the FAIR principles for the management of RWD generated by public funds (although they are also applicable to datasets created in the private sector) (Wilkinson 2016). Data sources should be Findable, Accessible, Interoperable, and Reusable. These recommendations are complemented by the recommendations of the Duke-Margolis white paper “Determining Real-World Data’s Fitness for Use and the Role of Reliability” (Mahendraratnam et al. 2019) that explored whether RWD are fit-for-purpose by the application of rigorous verification checks of data integrity.

While experts in modern validity theory have not reached consensus on the attributes of validity, there are basic tenets that most modern validity theorists have adopted (Royal 2017). Validity pertains to the inferences or interpretations made about a set of scores, measures, or in this case – data sources, as opposed to their intrinsic properties. As applied to evaluation of RWD sources, this means that they must be considered fit-for-purpose for generating credible RWE through well-designed and well-executed study protocols to inform decision making. Modern validity theory would suggest that the accumulation of evidence should be employed to determine if this inference regarding RWD quality is adequately supported. Hence, validity of a data source is a judgement on a continuum onto which new evidence is added and is assessed as part of a cumulative process because knowledge of multiple factors (e.g., new populations/samples of participants, differing contexts, new knowledge, etc.) are gained over time. This element of RWD source evaluation is not specifically recognized in the current recommendations by the FDA and the EMA.

One obstacle to developing a consensus regarding evaluation of data quality is that many terms have been used to describe their dimensions and elements, and the terminology has been used inconsistently, despite efforts at harmonization (Kahn et al. 2016). Despite this, RWE derived from RWD that focuses on the natural history of disease and adverse effects of treatment have long been considered “valid” by decision-makers. Recently, the issue of data validity has become more urgent and has been a focus for regulatory initiatives as RWE derived from RWD is being expanded in its use to inform decisions about treatment effectiveness and comparative effectiveness. These decisions demand a greater level of confidence and certainty in study results.

A crucial dimension in assessing the validity of data is the need for transparency and traceability/accessibility. This has been reinforced by the FDA in several recent draft guidance documents and the HMA-EMA (European Union’s Heads of Medicines Agencies-European Medicines Agency) Joint Big Data Taskforce Report (HMA-EMA 2019). The FDA guidance on Considerations for the Use of Real-World Data and Real-World Evidence to Support Regulatory Decision-Making for Drug and Biological Products states “If certain RWD are owned and controlled by third parties, sponsors should have agreements in place with those parties to ensure that all relevant patient-level data can be provided to FDA and that source data necessary to verify the RWD are made available for inspection as applicable” (FDA 2021b). The FDA noted in its Data Standards for Drug and Biological Product Submissions Containing Real-World Data Guidance for Industry that “during data curation and data transformation, adequate processes should be in place to increase confidence in the resultant data. Documentation of these processes may include but are not limited to electronic documentation (i.e., metadata-driven audit trails, quality control procedures, etc.) of data additions, deletions, or alterations from the source data system to the final study analytic data set(s)” (FDA 2021c).

Interest in the creation of “regulatory-grade” RWD/RWE has been spurred by the 21st Century Cures Act in the US and by the ongoing initiatives in Europe including the Innovative Medicines Initiative (IMI) Get Real and the HMA-EMA Big Data Joint Taskforce. As noted in the Framework for FDA’s Real-World Evidence Program to evaluate the potential use of RWE for the support of a new indication for a drug already approved or to help satisfy drug post approval study requirements, the strength of RWE submitted in support of a regulatory decision will depend on its reliability that encompasses more than transparency in data accrual and quality control, but also clinical study methodology, and the relevance of the underlying data (FDA 2018).

The European Medicines Regulatory Network strategy to 2025 includes the creation of the DARWIN [Data Analytics and Real-World Interrogation Network] (Arlett,2020, Arlett et al. 2021); it builds on the observation HMA - EMA Big Data Joint Taskforce Report (HMA-EMA 2019) that RWD is challenged by a lack of standardization, sometimes limited precision and robustness of measurements, missing data, variability in content and measurement processes, unknown quality and constantly changing datasets. The report viewed the number of European databases that currently meet minimum regulatory requirements for content and that are readily accessible, citing Pacurariu et al. (2018), as “disappointingly low”. The International Coalition of Medicine Regulatory Authorities (ICMRA) has called for global regulators to collaborate on standards for incorporating real-world evidence in decision making (ICMRA 2022).

In developing the screening tool, we attempted to find the right balance between the granularity of requested information and the response burden. We defined the dimensions of data suitability (e.g., sufficiency of quality and fitness-for-purpose) in plain English terms consistent with existing frameworks as discussed above. Although we focus on the US and EU, the tool may have relevance to other jurisdictions as well.

Since the authors have extensive experience with a wide variety of real-world data sources, the literature was assessed through a “snowballing” approach starting off with the identification of key references. This was supplemented by English language google searches applying the terms “real-world data” and “real-world evidence” and searches that combined these terms with variations on the terms “regulatory authorities”, “FDA”, and “EMA”. Additional searches combined via Boolean operators the terms “real-world data” with terms including “quality”, “validity”, and “verification”. FDA and EMA websites were also accessed.

The literature was summarized in tables to delineate the terms and concepts employed to assess the reliability (or quality) and relevance (whether it was fit-for-purpose for a proposed RWD study). We consolidated the concepts into 5 dimensions that, based upon our experience, conform to the general issues that must be clarified for most researchers to evaluate potential RWD sources.

The framework of the screening tool included the following dimensions: authenticity, transparency, relevance, accuracy, and track-record. We defined these dimensions as follows:

Authenticity: A data source is considered authentic when its provenance is well-documented, and its authenticity can be verified with its component data sources as needed.
Transparency: A data source is considered transparent when the processes used in data acquisition, curation, editing, and linkage are described adequately.
Relevance: A data source is considered relevant when it contains the population of interest in adequate numbers and sufficient length of follow-up; and it contains the data elements required in order to implement an analysis plan of a real-world protocol.
Accuracy: A data source is operationally considered accurate when it can be documented that it depicts the concepts they are intended to represent, insofar as can be assured by data quality checks (i.e., the data is of sufficient quality to investigate specific research questions). Further, key data elements can be audited as deemed required.
Track-Record: A data source will be considered more trustworthy in general when it can document its track-record of use cases that resulted in the creation of credible real-world evidence.

Using this framework, we developed the ATRAcTR (Authentic Transparent Relevant Accurate Track-Record) screening tool to evaluate real-world data sources (Appendix A). The component concepts of each of these dimensions were delineated in preparation for the creation of ATRAcTR (Table 1).

Table 1: Dimensions and Concepts Captured in ATRAcTR

Dimension	Concept
A. Authenticity	Provenance documentation Ability to verify provenance through data access Changes in data collection or sources over time Data Privacy Protection
B. Transparency	Data Extraction, Transformation, and Loading Handling of incorrect or missing data Handling of null values Data linkage Handling of free text Handling of changes in coding conventions Data latency and refresh rate Inclusion of synthetic data
C. Relevancy	Sample size of population of interest Representativeness Length of follow-up Demographic characteristics
D. Accuracy	Data checks including completeness, plausibility, conformance, uniqueness of patient records, and continuity of data collection Ability to audit key variables
Track Record	Historical performance of dataset in creating credible real-world evidence Has the data source been used to answer regulatory questions or support regulatory decision making

Authenticity is a foundational and necessary requirement for considering using an RWD source in real world evidence clinical study. This is consistent with FDA and EMA criteria requesting confirmation of data provenance and data traceability. While this may be obvious, recent experience during the COVID pandemic indicates that it cannot be taken for granted (Ledford and Van Noorden 2020). For some commercially available data sources, documentation alone may not be considered adequate. Verification via access to the data may be expected by regulators. Provenance documentation should provide information on what data was collected, why was it collected, the sources of the data, how the data was collected and the timeframe over which it was (or continues to be) collected. Any changes to the database over time should be disclosed (ex. addition of laboratory results, biomarkers, etc.). Disclosure is important to demonstrate that data has been appropriately de-identified and/or describe the procedures that are in place to protect patient confidentiality. Of course, the provenance of publicly supported datasets may be directly known by regulatory agencies via their participation in their creations (ex. SENTINEL and FDA, DARWIN and EMA) (Arlett 2020; FDA 2020; EMA 2021).

Transparency in the description of the processes and procedures employed to acquire, curate, transform, edit, and link data is essential to assessing whether planned analyses will be performed on reliable good quality data. Documentation should describe what ETL (extract, transform, load) procedure was used, whether the data were transformed according to specified standards into specified formats (ex. a common data model), how the data dictionary can be accessed, and whether there were any imputations or adjustments for incorrect or missing data. If more than one data source were combined to create the dataset, the processes for linking them should be described (ex. unique patient ID, tokenization). If the source data contained free-text contents, the processes for extracting information from the free text should be described (e.g., manually, natural language processing). How changes in coding conventions (ex. ICD-9 to ICD-10) over time were handled should also be described. For data sources that are still adding new data, the latency of data accrual and the refresh cadence should be described. Furthermore, whether the data sources contain any synthetic data should be disclosed including the reason for its inclusion.

The relevance of the data source is essential in determining whether a data source is fit-for-purpose and is a bespoke process for a particular real-world clinical study. It includes assessing the sample size of the population of interest it contains, the observation period and length of follow-up. Documentation must be provided to allow researchers to assess whether data elements required for analysis are captured. Additional useful information is provided by demographic characteristics of the data including age, gender, ethnicity, and geography in the dataset. How this request is interpreted (e.g., overall database, study population) will depend on how the data source provider understands the needs of the researcher. The interpretation of data source representativeness will depend on its intended use. It is useful to disclose whether the data source was designed to be representative of the population, or if it is a convenience sample, and it should be stated why it can represent the population of interest.

We define accuracy broadly here as an assessment of the integrity of the data, most frequently characterized by conducting data quality checks for conformance to a common data model, plausibility and completeness; it also includes the concepts of reliability, extensiveness, and coherence as discussed in the EU framework. We separated accuracy from transparency since this requires its own distinct review procedures while recognizing that both are components of reliability and in deciding whether a data source is fit-for-purpose. A substantial literature (Kahn et al. 2016; Blacketer et al. 2021; Dreyer et al. 2010; Dreyer 2018; Girman et al. 2019; Hall et al. 2012; Kahn et al. 2015; Liaw et al. 2021; Miksad and Abernathy 2018; Qualls et al. 2018; Razzaghi et al. 2022; Reynolds et al. 2020; Schmidt et al. 2021; Simon et al. 2022; Zozus et al. 2015) describes the need for data quality checks and their specifics. Individual data curators generally set their own standards, making disclosure important. Because of the longer experience with administrative claims data, there are more generally accepted standards than there currently exist for EHR data. Software programs (ex. Open-source R or Python packages or functions, open source and commercial data quality assessment tools) have been developed to make the process operationally feasible (Liaw et al. 2021; Schmidt et al. 2021). Plausibility checks include an assessment of external consistency and internal consistency (e.g., temporal value violations, unexpected distributions or combinations, out-of-range value anomalies, data contradictions, and how null values were handled). There should also be disclosed whether there were data quality checks performed to evaluate if any unique patient contributed data under more than one identifier. Moreover, it should be disclosed whether there was any interruption or significant changes in data collection and/or processing over time that impacted data continuity (e.g., who contributed data, what data was collected, or how it was processed).

As discussed earlier, and consistent with Modern Validity Theory, one gains insight into the general quality of a data source based upon its performance track-record of use cases where a data source successfully or failed to result in credible RWE. Performance track-record would include the publications that have employed the data source (especially for similar research questions to the study of interest) as well as a listing of unpublished use cases. If the data source has been used in a study designed to emulate a randomized controlled trial (Franklin et al. 2021) and was successful, this may provide additional confidence in the trustworthiness of the dataset. Use cases and/or publications that were incorporated in responses to regulatory questions or to support regulatory decisions should be highlighted. For newly available data sources, previous performance history may be meager; however, such datasets can be provisionally viewed as suitable based on their providing evidence that they satisfy the criteria of authenticity, transparency, relevancy, and accuracy. As noted earlier, examining the track records of RWD sources has not been explicitly identified in current frameworks; we suspect that this reflects the fact that evaluating relevance or fitness-for-purpose is a bespoke judgement relating to the particular decision facing a regulator.

At first glance, the response burden for this tool may appear to be somewhat overwhelming. However, we assessed the burden by creating a set of prototype responses to the tool for a hypothetical claims data source (Appendix B), and it was not an undue burden when you have the appropriate knowledge. Note that, for a given data source, the information requested remains the same across different studies. Appendix B offers an example of what is a reasonable granularity of detail that provides a good picture of data quality without requiring an exhaustive response burden. We expect that whoever responds to the tool will determine what is reasonable in discussion with researchers seeking to obtain access to the data source.

Regulatory bodies such as the FDA and EMA, as well as others, have turned their attention to Real-World Evidence to complement the information from randomized controlled trials to assess the benefit-risk profiles of therapeutics. They have long used evidence derived from real-world data sources to assess safety profiles and have begun moving from passive to active surveillance through the use of networks such as SENTINEL. Increasingly, standing cohorts are being leveraged for signal detection and confirmation (Huybrechts et al. 2021).

On a limited basis, RWE has been considered adequate to support labelling changes when randomized controlled trials have not been considered feasible (FDA 2021a). Recently, RWE has been accepted by the FDA as the sole supplementary information to support labeling changes (FDA 2021d). Tacrolimus was approved for prophylaxis of organ rejection in patients receiving liver transplants based upon a Supplemental New Drug Application supported by a non-interventional RWE study. Tacrolimus had been approved for use in kidney and heart transplants. The data source used in the RWE study was the U.S. Scientific Registry of Transplant Recipients. There were minimal concerns regarding relevance since the registry contained data on all lung transplants in the U.S., death and graft failures were adjudicated, and most necessary information was captured. There were low concerns on reliability since there was transparent RWD management, there was a low percentage of missing variables and few non-plausible lengths of hospitalization; most variables were coded accurately. The FDA determined that this non-interventional study with historical controls to be adequate and well-controlled and noted that, outcomes of organ rejection and death are virtually certain without therapy. The dramatic effect of treatment helped to preclude bias as explanation of results.

With the widespread digitization of medical information, how to expand the use of real-world data to assess clinical effectiveness has become a priority both in the U.S. and in Europe. Indeed, the FDA has utilized RWE on a limited basis to evaluate the effectiveness of treatments for rare diseases and cancers. However, RWD submitted to support effectiveness claims have been frequently found deficient due to issues of relevancy (ex. representativeness of population, small sample size) and accuracy (e.g., missing data) (Mahendraratnam et al. 2022; Bakker et al 2022). Publicly funded efforts to create “fit-for-purpose” RWD have begun in both the U.S. and Europe including SENTINEL, PCORNet, and DARWIN. Use of SENTINEL has been restricted thus far to safety issues but its extension to its application in treatment effectiveness studies is on the table as part of the FDA Real-World Evidence Program. Progress in this endeavor will depend on changing the culture, behaviors, and standards applied by researchers and data source providers; in turn, the impetus for such change will come, in large part, by changing requirements of regulatory bodies as they expand the use of RWE to address perceived weaknesses of real-world observational studies in comparison to randomized controlled trials (Berger and Crown 2022; Simon et al. 2022).

Currently, for commercially available data sources, there is limited disclosure (e.g., transparency) describing how datasets are assembled, nor is there much disclosure and/or published literature on assessments of their data quality. Regulators and researchers have had to rely on their experience with specific data sources to judge their reliability rather than a formal set of criteria. For data sources that are new to the regulators and researchers, or for researchers with less experience or knowledge of RWD, it may be difficult to judge the quality of the data sources without sufficient disclosure from the data providers.

The screening tool described here will be immediately helpful for researchers evaluating RWD sources to gain understanding of the data provenance and how patient confidentiality is protected, how data are extracted and curated, what data quality checks have been performed, and what has been the performance history of the data source. It is not expected that all data source providers will be necessarily willing or able to provide all of the information requested in ATRAcTR due to concerns regarding protection of intellectual property. However, the greater number and more granular the information they are able to provide will increase confidence in those requesting access to the dataset that it has the potential to generate regulatory-grade RWE. We note that the REPEAT Initiative (Wang et al. 2022) has been able to reproduce the results of a large number of published comparative safety or effectiveness studies employing several commonly accessed real-world data sources (e.g., Medicare, Marketscan, Optum, and CPRD); this suggests that the providers of these data sources are likely be able to provide complete and adequate responses to the ATRAcTR.

We believe that the five dimensions of ATRAcTR represents a logical and intuitive approach for researchers with a range of experience in using RWD sources and that they are consistent with the current advice being provided by the FDA and EMA. We expect that as criteria for evaluating RWD moves toward international harmonization that the information sought in our tool will be captured in any future recommendations.

There are some limitations in our efforts to develop the ATRAcTR. We did not use a formal process to delineate its dimensions since our framework is compatible with existing conceptual frameworks, with the exception of our addition of reviewing the track record for a particular RWD source. Neither did we perform a formal validation study, since this is intended to be a screening instrument and researchers will clearly be in a better position with this information. In the end, they will have to judge for themselves how to interpret the information received. In addition, we did not try to establish what are minimum standards to conclude that a specific RWD source is of “good quality” and “fit-for-purpose”. Nevertheless, we are confident that the ATRAcTR can be helpful to researchers in identifying potential regulatory-grade RWD sources.

Acknowledgments: The authors appreciate the editorial support from Arghya Bhattacharya,PhD and Shanthakumar V, PhD of Viatris. The abstract was presented at the 2023 International Conference on Health Policy Statistics.

Arlett, P., DARWIN EU (Data Analytics and Real World Interrogation Network), September 2020. Available online: https://www.ema.europa.eu/en/documents/presentation/presentation-proposal-darwin-eu-data-analytics-real-world-interrogation-network-parlett-ema_en.pdf (Accessed on May 31, 2023)
Arlett, P., Kjaer, J., Broich, K., Cooke, E., Real-world evidence in EU Medicines regulation: Enabling Use and Establishing Value. Clinical Pharmacology and Therapeutics 111(1), 21-23 (2021)
Bakker, E., Plueschke, K., Jonker, C., Kurtz, X, et al. Contributions of Real-World Evidence in European Agency’s Regulatory Decision Making. Clinical Pharmacology and Therapeutics 113(1), 135-151 (2022)
Berger M.L., Axelsen, K., Lipset, C., Gutteridge, A. et al. Optimizing the Leveraging of Real World Data: How It Can Improve the Development and Use of Medicines. Value in Health, 18, 127-130 (2015)
Berger, M.L., Crown, W. How can we make more rapid progress in the leveraging of real-world evidence by regulatory decision makers? Value in Health 25(20), 167-170 (2022)
Berger, M.L., Daniel, G., Frank, K., Hernandez, A., McClellan, M., et al. A Framework for Regulatory Use of Real-World Evidence. Duke-Margolis Center for Health Policy, September 13, 2017. Available online: https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf (Accessed on May 31, 2023)
Blacketer, C., Defalco, F., Ryan, P., Rijnbeek, P. Increasing trust in real-world evidence through evaluation of observational data quality. Journal of the American Medical Informatics Association, 28(10), 2251–2257 (2021)
Daniel, G., Silcox, C., Bryan, J., McClellan, M., et al. Characterizing RWD Quality and Relevancy for Regulatory Purposes. Duke Margolis Center for Health Policy, October 1, 2018. Available online: https://healthpolicy.duke.edu/publications/characterizing-rwd-quality-and-relevancy-regulatory-purposes-0 (Accessed on May 31, 2023)
Dreyer, N., Advancing a Framework for Regulatory. Use of Real-World Evidence: When Real Is Reliable Therapeutic Innovation & Regulatory Science. 52(3), 362-36 (2018)
Dreyer, N., Schneeweiss, S., McNeil, B, Berger, M., et al. GRACE Principles: recognizing high-quality observational studies of comparative effectiveness. American Journal of Managed Care 16(6), 467-471 (2010)
European Medicines Agencies / Heads of Medicines Agencies, Data Quality Framework for EU medicines regulation. September 2022. Available online: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/data-quality-framework-eu-medicines-regulation_en.pdf (Accessed May 31, 2023)
European Medicines Agency, EU Big Data Stakeholder Forum, December 7, 2021 Available online: https://www.ema.europa.eu/en/documents/report/report-eu-big-data-stakeholder-forum-2021_en.pdf (Accessed on May 31, 2023)
FDA, Framework for FDA’s Real-World Evidence Program, December 2018 Available online: https://www.fda.gov/media/120060/download (Accessed on May 31, 2023)
FDA, FDA and the Sentinel Operations Center, Standardization and Querying of Data Quality Metrics and Characteristics for Electronic Health Data, Data Quality Metrics System Final Report, Version # 5.1, Sentinel Initiative, September 15, 2020. Available online: https://www.sentinelinitiative.org/sites/default/files/Methods/Standardization_and_Querying_of_Data_Quality_Metrics.pdf (Accessed on May 31, 2023)
FDA (2021a), Considerations for the Use of Real-World Data and Real-World Evidence to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry, December 2021 Available online: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-real-world-data-and-real-world-evidence-support-regulatory-decision-making-drug (Accessed on May 31, 2023)
FDA (2021b). Assessing Electronic Health Record and Medical Claims Data to Support Regulatory Decision Making. September 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/real-world-data-assessing-electronic-health-records-and-medical-claims-data-support-regulatory (Accessed, May 31, 2023)
FDA (2021c), Data Standards for Drug and Biological Product Submissions Containing Real-World Data Guidance for Industry, October 2021. Available online: https://www.fda.gov/media/153341/download (Accessed on May 31, 2023)
FDA (2021d). FDA approves new use of transplant drug based on real-world evidence – 7/16/21. (https://www.fda.gov/drugs/news-events-human-drugs/fda-approves-new-use-transplant-drug-based-real-world-evidence) (accessed May 31, 2023)
FDA, Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders, June 2022. Available online: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/patient-focused-drug-development-selecting-developing-or-modifying-fit-purpose-clinical-outcome (Accessed on May 31, 2023)
Franklin J., Patorno, E., Desai, R., Glynn, R., et al. Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies: First Results From the RCT DUPLICATE Initiative. Circulation 143(10), 1002-1013 (2021)
Girman, C., Ritchey, M., Zhou, W., Dreyer, N. Considerations in characterizing real‐world data relevance and quality for regulatory purposes: A commentary. Pharmacoepidemiol Drug Saf. 28, 439–442 (2019)
Hall, G. Sauer, B., Bourke, A., Brown, J., et al. Guidelines for Good Database Selection and use in Pharmacoepidemiology Research. Pharmacoepidemiology and drug safety 21, 1–10 (2012)
HMA-EMA Joint Big Data Taskforce Phase II Report: Evolving Data-Driven Regulation 2019. Available online: https://www.ema.europa.eu/en/documents/other/hma-ema-joint-big-data-taskforce-phase-ii-report-evolving-data-driven-regulation_en.pdf (Accessed on May 31, 2023)
Huybrechts, K., Kulldorff, M., Hernández-Díaz, S., Bateman, B., et al. Active Surveillance of the Safety of Medications Used During Pregnancy. American Journal of Epidemiology 190(6), 1159–1168 (2021)
International Coalition of Medicine Regulatory Authorities, ICMRA Statement on International Collaboration to enable real world evidence (RWE) for regulatory decision-making. Available online: https://icmra.info/drupal/sites/default/files/2022-07/icmra_statement_on_rwe.pdf (Accessed on May 31, 2023)
Kahn, M., Brown, J., Chun, A., Davidson, B., et al. Transparent Reporting of Data Quality in Distributed Data Networks. eGEMs (Generating Evidence & Methods to improve patient outcomes, 3(1):7 (2015) Available online: http://dx.doi.org/10.13063/2327-9214.1052 (Accessed on May 31, 2023)
Kahn, M., Callahan, T., Barnard, J., Bauck, A., et al. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data," eGEMs (Generating Evidence & Methods to improve patient outcomes).4: (1) 18 (2016) Available online: http://dx.doi.org/10.13063/2327-9214.1244 (Accessed on May 31, 2023)
Ledford H, Van Noorden R. High-profile coronavirus retractions raise concerns about data oversight. Nature 582(7811), 160 (2020)
Liaw. S., Guan, J., Guo, N., Ansari, S., Quality assessment of real-world data repositories across the data life cycle: A literature review. Journal of the American Informatics Association, 28(7), 1591-1599 (2021)
Mahendraratnam, N., Mercon, K., Gill, M., Benzing, L, McClellan, M., Understanding Use of Real-World Data and Real-World Evidence to Support Regulatory Decisions on Medical Product Effectiveness. Clinical Pharmacology & Therapeutics 111(1), 150-154 (2022)
Mahendraratnam, N., Silcox, C., Mercon, K., Kroetsch, A., et al. Determining Real-World Data’s Fitness for Use and the Role of Reliability. Duke Margolis Center for Health Policy. September 26, 2019. Available online: https://healthpolicy.duke.edu/sites/default/files/2019-11/rwd_reliability.pdf (Accessed on May 31, 2023)
Miksad R., Abernethy, A. Harnessing the Power of Real-World Evidence (RWE): A Checklist to Ensure Regulatory-Grade Data Quality. Clinical Pharmacology and Therapeutics. 103(2): 202-205 (2018)
Pacurariu, A., Plueschke, K., McGettigan, P., Morales, D., et al. Electronic healthcare databases in Europe: descriptive analysis of characteristics and potential for use in medicines regulation, BMJ Open 8, e023090. doi:10.1136/bmjopen-2018-023090 (2018)
Qualls L., Phillips T., Hammill B., Topping J. et al. Evaluating Foundational Data Quality in the National Patient-Centered Clinical Research Network (PCORnet®). EGEMS (Wash DC). 2018 Apr 13, 6(1), 3. doi: 10.5334/egems.199. PMID: 29881761; PMCID: PMC5983028.
Razzaghi, H., Greenberg, J., Bailey L., Developing a systematic approach to assessing data quality in secondary use of clinical data based on intended use. Learn Health Sys. 6:e,10264 (2022) https://doi.org/10.1002/lrh2.10264
Reynolds, M., Bourke, A., Dreyer, N. Considerations when evaluating real-world data quality in the context of fitness for purpose. Pharmacoepidemiol Drug Saf. 29, 1316–1318 (2020)
Royal, K., Four tenets of modern validity theory for medical education assessment and evaluation. Adv Med Educ Pract. 8, 567–570 (2017)
Schmidt, C., Struckmann, S., Enzenbach, C., Reineke, A., et al. Facilitating harmonized data quality assessments. A data quality framework for observational health research data collections with software implementations in R. BMC Medical Research Methodology 21, 63 https://doi.org/10.1186/s12874-021-01252-7 (2021)
Simon, G., Bindman, A., Dreyer, N., Platt, R., When Can We Trust Real-World Data To Evaluate New Medical Treatments? Clinical Pharmacology & Therapeutics 111(1), 24-29 (2022)
Toward the European Health Data Space (TEHDAS EU). European Health Data Space Data Quality Framework, Deliverable 6.1 of TEHDAS EU 3rd Health 566 Program (GA: 101035467). May 2022, Available online: https://tehdas.eu/results/tehdas-develops-data-quality-recommendations (Accessed May 31, 2023)
TransCelerate RWD Audit Readiness Considerations – Draft for Public Review December 2022. Available online: https://www.transceleratebiopharmainc.com/wp-content/uploads/2022/12/RWD-Audit-Readiness-Considerations-Document.pdf (Accessed May 31, 2023)
Wang, S., Sreedhara, S., Schneeweiss, S., Repeat Initiative, Reproducibility of real-world evidence using clinical practice data to inform regulatory and coverage decisions. Nature Communications 13, 5126 (2022) https://doi.org/10.1038/s41467-022-32310-3
Wilkinson, M. The FAIR Guiding Principles for scientific data management and stewardship. 2016. Scientific Data 3, 160018 (2016)
Zou, KH., Li, JZ., Salem, LA., Imperator, JS., et al. Harnessing real-world evidence to reduce the burden of noncommunicable disease: health information technology and innovation to generate insights. Health Serv Outcomes Res Methodol. 21(1), 8–20 (2021)
Zozus, M., Hammond, W., Green, B., Kahn, M., et al. Assessing Data Quality for Healthcare Systems Data Used in Clinical Research (Version 1.0). NIH Collaboratory, DOI: 10.13140 (2015) Available online: https://dcricollab.dcri.duke.edu/sites/NIHKR/KR/Assessing-data-quality_V1%200.pdf. (Accessed on May 31, 2023)

No competing interests reported.

Download PDF

Journal Publication

published 29 Nov, 2023

Read the published version in Health Services and Outcomes Research Methodology →

Editorial decision: Major revision
26 Aug, 2023
Reviews received at journal
25 Jul, 2023
Reviewers agreed at journal
18 Jun, 2023
Reviewers invited by journal
16 Jun, 2023
Editor assigned by journal
16 Jun, 2023
Submission checks completed at journal
05 Jun, 2023
First submitted to journal
05 Jun, 2023

You are reading this latest preprint version

ATRAcTR (Authentic Transparent Relevant Accurate Track-Record): A Screening Tool to Assess the Potential for Real-World Data Sources to Support Creation of Credible Real-World Evidence for Regulatory Decision-Making

Status:

Journal Publication

Version 1

Abstract

1. Introduction

2. Materials and Methods

3. Results

4. Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1