The ACT NOW CE Study was a multi-site, retrospective chart review conducted between July 1, 2016 and June 30, 2017 “to inform the design of a clinical trial to improve care and outcomes for infants with neonatal opioid withdrawal syndrome (NOWS)”.16 Thirty sites from the Environmental influences on Child Health Outcomes IDeA States Pediatric Clinical Trials Network (ECHO ISPCTN) and the Eunice Kennedy Shriver National Institute of Child Health and Human Development’s (NICHD) Neonatal Research Network (NRN) distributed across the U.S. participated in the study. The medical records of approximately 1,800 infants with NOWS were abstracted across all sites, of which a subset of cases (over 200) underwent a formalized QC process to identify data quality errors and determine the association between MRA and data quality.
To evaluate the MRA process, continuous QC monitoring was performed throughout the course of the ACT NOW CE Study. This process required a certain percentage of cases at each site to be re-abstracted by a second, independent abstractor from that site. Prior to the start of the ACT NOW CE Study, (1) the acceptable error rate threshold was set (no greater than 4.93% or less than 500 errors per 10,000 fields),15 (2) a formal abstraction guideline for the study was developed to ensure consistency in data collection across abstractors and sites, and (3) each abstractor (primary abstractor and QC-abstractor) received extensive MRA and QC training. At a minimum, each site performed QC on the first 3 cases abstracted by the site (QC1). Depending on the total number of cases abstracted by the site, additional QC “events” would be required after every 25 cases, one randomly selected case for every 25 cases abstracted and entered into the electronic data capture (EDC) system (QC25, QC50, etc.). Accordingly, the total number of QC Events conducted per site corresponded to the total number of cases abstracted at the site. Seven distinct QC Events were observed over the course of the study (QC1, QC25, QC50, QC75, QC100, QC125, QC150).
A high-level overview of the QC process is described here. The primary abstractor would perform MRA on a set of cases (up to 3 for QC1 and an additional 1 case for every QC Event thereafter). The site would notify the Data Coordinating and Operations Center (DCOC) once the specified number of cases had been entered into the EDC and cease all data collection and entry until QC was completed. Using a random number generator, the DCOC identified case(s) for QC and notified the site’s QC-abstractor, who would independently abstract the assigned case(s). The QC-abstractor was not able to see how the primary abstractor identified the data elements within the EHR. Essentially, the QC-abstractor carried out their abstraction and data entry as if it was a completely new case. Upon completion, an automated script was triggered to run, which compared the data entered for each QC case (primary- vs. QC-abstractor) and generated a report with a list of discrepancies.
The system considered any inconsistency between the primary- and the QC-abstractor as a discrepancy. By design, the system was highly sensitive to detect any inconsistency in data entry. Once the report was generated, an informaticist and site manager from the DCOC met with the site (both the primary- and QC-abstractors) via video conference, and reviewed the results of the discrepancy report. During the review, sites referenced their EHRs to identify the true value for all discrepancies. The team reviewed each discrepancy and identified true errors. A discrepancy was considered a true error if the primary abstractor had entered data into the EDC that was inconsistent with what was in the EHR (the gold standard). If the primary-abstractor’s data entry matched the EHR, the discrepancy was not considered a true error (even if the QC-abstractor did not match). The error rate was then calculated and shared with the site along with a corrective action plan.
In the event that a site exceeded the acceptance criteria for the specified QC Event, the site would be required to repeat the event on another 3, randomly selected cases. Cases for repeat QC Events were selected by the DCOC as follows. First, the DCOC identified the treatment type (pharmacologic vs. non-pharmacologic), referred to as the case type, most prominent in the current set of abstracted cases at the site, and, then, randomly selected 2 cases of the most prominent case type and 1 case from the other case type for re-abstraction. In situations where the site exceeded the acceptance criteria for the repeat QC, the site (primary- and QC-abstractors) would be required to participate in retraining and perform (and pass) another QC before continuing with the study. If the site was within the acceptable limits, they were able to continue with data collection.
The Error Rate Calculation framework, outlined in the Good Clinical Data Management Practices (GCDMP) guidelines,17 was used to describe error rates, the distribution of the error rates, and the error rates over time. Simply put, error rate is a ratio between the number of data errors detected compared to the total number of data fields collected:
(1)
For this study, we initially calculated the crude MRA error rates along with the Wald’s 95% confidence intervals (CIs) over time. Error rates were calculated using all/total fields (“all-field” error rate) and using only populated fields (“populated-field” error rate) to provide both an optimistic and a conservative measurement, respectively, to account for the variability in the calculation and reporting of error rates in the literature. We derived an adjusted MRA error rate along with a 95% CI using a generalized estimating equation model to account for the clustering.
Of the 1,808 total cases, 219 cases were selected for QC. Four of those cases did not have QC performed due to data entry issues that did not allow for a full QC report to be generated. These were excluded from the analysis. Thus, the analytic sample consisted of 215 QC cases. When calculating the error rates for both the all-field and populated-field totals, the study population was divided into two groups, or case types, based on the methods used to treat the infant for NOWS: using pharmacologic therapy (P) or using only non-pharmacologic therapies (NP). The number and types of data elements varied between the two groups (the pharmacologic treatment group requiring more variables), differences that could cause variation in the number of errors. After reviewing the adjusted error rates by case type, the decision was made to combine cases when calculating the changes in error rates over time, as the adjusted error rates by case type did not offer statistically significant results to warrant further investigation. We derived both the crude and adjusted populated-field error rates at each of the 7 distinct QC Event times. For each set of crude and adjusted populated-field error estimates, we fitted separate time series regression with a time trend as the independent variable. We report both the slope estimates and corresponding 95% confidence limits.
The overall error rates for the ACT NOW CE Study were also compared to error rates from the literature to assess the impact formalized MRA training and continuous QC monitoring could have on data quality in clinical research. Prior work conducted by Zozus and colleagues3,4 synthesized the existing data quality literature across various clinical research studies. Quantitative data accuracy information was abstracted from the articles and pooled. Manuscripts were categorized by type of secondary data use, data processing method, and data accuracy assessment. Information on the number of errors identified and the number of fields inspected was collected for each manuscript. From this work, we referenced 71 MRA-centric studies to conduct a meta-analysis of the overall error rate as reported in the existing literature for comparison against the ACT NOW CE Study. Based on the residual and leave-one-out diagnostics, we identified 5 studies that were deemed to be potential outliers. Thus, these studies were removed for the final meta-analysis to obtain the estimate error rate for the literature reviews.
In order to derive an overall MRA error rate for comparison with the current study, we performed a meta-analysis of single proportions to derive an overall error rate from the literature based on an inverse variance method and generalized linear mixed model approach using the R package “metafor”.18 Additionally, we analyzed the data from each site within the ACT NOW CE Study separately to obtain site-specific error estimates and used meta-analysis to obtain a single common error estimate.19 Next, we compared the all-field and populated-field error rates from our study with the error estimate derived from the literature based on independent meta-analyses using a Wald-type test statistic. As a sensitivity analysis, we also compared the error rate estimate from the meta-analysis with our study estimate based on a meta-regression model with the inclusion of an indicator moderator to distinguish studies from the literature with our study. The meta-analysis was limited to a comparison in overall error rates, as the studies identified in the literature did not provide the level of granularity to support a comparison by data or case type.