Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

doi:10.21203/rs.3.rs-1496802/v1

Download PDF

Research Article

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

https://doi.org/10.21203/rs.3.rs-1496802/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Papers describing the results of a randomised trial should include a table of summary statistics that compares randomised groups at baseline. Researchers who fraudulently generate trials often create randomised groups that are implausibly similar (under-dispersed) or accidentally create large differences between groups (over-dispersed) because they do not understand how to create realistic summary statistics from truly random data. We aimed to automatically screen for under- and over-dispersion in the baseline tables of published randomised trials.

Methods: We examined randomised controlled trials published in open access health and medical journals on PubMed Central. We estimated probability that a trial’s summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences from the baseline table, and compared this with an expected distribution without dispersion. We used a simulation study to test the ability of the model to find over- or under-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values.

Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, as it had a higher probability of detecting over-dispersed data and lower false positive percentage for skewed data that was not under- or over-dispersed. For the trials published on PubMed Central, some tables were flagged because: they were not from trials, had a non-standard presentation, or had likely reporting errors. The algorithm also flagged some trials as under-dispersed where there was a striking similarity between groups.

Conclusion: Due to the variance in reporting across journals, the current method may not be suitable for automated screening for fraud of all submitted trials, but could be useful in targeted checks of suspected trials. If used in automated screening of papers at the submission stage, it would likely flag many instances of non-standard table presentation and poor reporting.

randomised controlled trials

research misconduct

reporting errors

Papers describing the results of a randomised trial often include a table that compares the randomised groups at baseline (hereafter called a “baseline table”). This baseline table presents summary statistics that describe the sample, such as average age and the percentage of males. The table’s purpose is to demonstrate that the randomisation produced similar groups, which strengthens the case that any differences between groups are due to the randomised treatment [1]. A baseline table is recommended by the CONSORT guidelines, which were designed to improve the reporting of randomised trials [2].

If randomisation has been successful, then the summary statistics will be similar across all groups at baseline, although large differences can happen simply due to chance, especially for small sample sizes. To convey similarity many authors compare the randomised groups using multiple statistical tests, despite the circularity of testing if randomisation happened by chance [3].

Researchers who fabricated randomised trials have been discovered because their baseline tables were too perfect [4–6]. When fabricating the baseline table they created highly comparable groups that would pass peer review. In not trying to raise any alarms during peer review, they unwittingly raised the alarm at post peer review. Fraudulent researchers can also create baseline data with unusually large differences between groups, likely because they do not understand how to create realistic summary statistics when data are truly random [7]. Fraudulent researchers may not be uncovered by one baseline table alone, but an odd table might prompt a wider investigation [8].

To date, fraudulent researchers have been found in ad hoc ways, including concerns being raised by whistle-blowers and researchers noticing strange patterns whilst reading papers in their field or conducting systematic reviews [9, 10]. Other problems have been found by dedicated researchers trawling through papers by hand. Manually extracting data from papers is time consuming and automatic data extraction would be a useful advance. Automated detection tools would save time and increase scrutiny [11].

Previous methods for spotting potential problems in baseline tables have used p-values for the tests comparing groups at baseline, and then tested if the distribution of p-values was uniform [12]. However, it is possible to get a non-uniform distribution of p-values when the two groups were randomised, for example, for skewed data [13]. Another limitation with this approach is that it can only use summary statistics of the mean and standard deviation for continuous variables, which means that percentages are lost [14, 15]. This is a large loss as percentages are commonly used in baseline tables. In this paper we avoid the issues of using the distribution of p-values by instead examining the distribution of t-statistics. We test whether the distribution of t-statistics is under- or over-dispersed compared with a typical randomised trial.

The aim of this paper is not to provide undeniable evidence for fraud. Baseline tables that appear to have a problem could occur for a range of non-fraudulent reasons. These include planned or unplanned factors to do with randomisation, such as dynamic randomisation to create highly comparable groups, or subversion of the random allocation [16]. Differences could also arise at the reporting stage, such as mislabelling summary statistics or reporting errors [11]. Our aim is to create a procedure that could be used to flag potential problems at the submission stage, which would help researchers improve their paper prior to publication [17].

We report PubMed Central ID numbers to highlight examples without citing papers. The example tables can easily be examined by interested readers using the PubMed Central site (see Additional file 1).

Automated extraction of baseline tables

We extracted baseline tables from the National Library of Medicine’s PubMed Central open access subset which has 3.7 million papers. The data were downloaded on 12 January 2022. The basic steps were:

Download a list of randomised trials from the trialstreamer web page [18] using the Pubmed Central ID (PMCID). The trialstreamer data was downloaded on 9 August 2021 and had 57,109 trials with a PMCID.
Make a random selection of 10,000 trials.
Access the available papers from the open access subset using the PMCID; download the paper in XML format and read it into R.
Exclude papers that were not randomised trials, including i) protocols which have no data, and ii) papers that re-use trial data for other study designs (e.g., diagnostic accuracy). This exclusion was made based on the title and abstract, but some protocols were not identified in the title or abstract and hence were wrongly included.
Find the baseline table in the paper and extract all the summary statistics, the type of summary statistics (e.g., median or percentage), and the group sample sizes.
A key step was estimating what summary statistics were used in each row of the baseline table. This was estimated based on the text in the rows and columns (e.g., “Mean”, “%”, etc) and the variable label, where particular variables such as age and BMI were often continuous, whereas variables such as gender were categorical.

Another key step was estimating the group sample size. These were first estimated by checking for headings in the column headers (e.g., “N=”) and then the rows. If no samples sizes could be found then they were estimated based on percentages in the baseline table. Some papers were excluded because no sample size could be extracted (see results).

We extracted whether p-values were used in the table and/or mentioned in the text. We searched for text such as ‘p-value’ and ‘statistically significant’, and searched for common thresholds such as ‘<0.05’. We searched the paragraph of text that first referenced the baseline table to check whether authors mentioned statistical significance.

Validation

We validated our algorithm using hand-entered baseline data from randomised trials. To find eligible trials we searched PubMed for randomised trials between 2017 and 2020 that were available open access on PubMed Central and were not protocols, which gave 25,760 trials. A random selection of 200 trials was made, with the results compared between the algorithm and manually entered data by a post-doctoral research assistant. 118 papers were excluded by the algorithm, with the three most common reasons because the paper was not openly available ($n=48$), there was no baseline table ($n=36$), or there was no comparison between groups in the table ($n=16$). A further 9 papers could not be compared because the research assistant judged them not to be randomised trials. This left 73 baseline tables to compare.

Detailed comparisons of the algorithm and manual results are given in Additional file 1. In summary, our algorithm correctly guessed the summary statistic 87% of the time (795 out of 909; 95% CI 85–90%). The biggest differences were where the algorithm wrongly guessed continuous for a median (2%) or could not estimate any statistic when the row was a percentage (5%). The algorithm was able to extract the sample size, with a mean difference of 0 (5–95% percentile: 0 to 0). The algorithm accurately estimated the size of the baseline table, with a median difference of 0 (5–95% percentile: − 1 to 4) for the number of rows, and 0 (5–95% percentile: 0 to 0) for the number of randomised groups.

We think this level of accuracy is reasonable as we encountered a large variety in the presentation of baseline tables, an issue that has been flagged by others [19]. Failures of the algorithm sometimes meant the table data were excluded as our approach struggled to extract the numbers, this means that we are not able to completely screen the literature. Failures where the wrong data were extracted, for example wrongly extracting a total column as if it were a separate group, sometimes led to the trial being flagged by our model and we examine this issue in the results.

We compared the total sample size in our algorithm with that estimated by the trialstreamer program [18] for the 1,684 trials which had estimated sample sizes from both approaches. For 71% of trials the difference was within $\pm$ 10%. Some of the larger differences occurred because trialstreamer estimated the sample size based on the abstract, whereas our algorithm used the baseline table and there were trials that only used a subset of the full data (e.g., PMC6746080). There were also occasions where trialstreamer extracted a year rather than the number randomised (e.g., PMC4852093), and where trialstreamer used the number of clusters in a cluster randomised trial whereas our algorithm used the number of participants (e.g., PMC7690135).

Creating t-statistics for group differences

We combined continuous and binary summary statistics from the baseline table by summarising the difference between randomised groups using the independent sample t-statistic, which put the group differences on a common scale. It may be surprising to use the t-statistic to compare binary data, but the t-test is robust in situations where the chi-squared test would be a common choice, even for small sample sizes [20]. Combining continuous and binary summary statistics greatly increases the amount of data we examined, as these two statistics were the most commonly used. When using p-values the recommendation is to only use continuous summary statistics [15]. We could not use summary statistics that were the median and quartiles or minimum to maximum, as we could not compare these statistics using the parametric t-test.

We excluded rows from the baseline table that were the inverse of the previous row, for example the percentage male followed by the percentage female. In this case the t-statistics for males and females would be perfectly negatively correlated and including these results twice would artificially increase the sample size. We excluded rows from the same trial where the t-statistic was the inverse of the previous, but not where the t-statistic was zero. This approach only excludes rows that are a perfect inverse and would miss results with a relatively large negative correlation, such rows for three age groups.

We created t-statistics for all pairs of comparisons. For example, a trial with three treatment arms would have three comparisons (A vs B, A vs C, B vs C). This potentially created correlation in the t-statistics, which we did not adjust for as the approach screens for overall in the baseline table, and in cases of fraud the problems may well appear for all groups.

Bayesian model of observed differences

The observed differences ($d$) in randomised groups were modelled using a t-distribution.

$${d}_{i,j}\sim t(0,{\sigma }_{i,j}^{2},{\text{df}}_{i}), i=1,\dots ,M,\hspace{0.17em}j=1,\dots ,{n}_{i},$$

where $i$ is the trial index and $j$ is the row in the table. The difference is either the difference in reported means for continuous statistics, or the difference in reported proportions for categorical statistics. The mean difference is zero which is as expected for randomised groups at baseline.

The pooled variance (${\sigma }_{i,j}^{2}$) either uses the reported standard deviations for continuous statistics or reported proportions for categorical. The pooled inverse-variance was modelled as

$${\sigma }_{i,j}^{-2}={s}_{i,j}^{-2}\times {\gamma }_{i},$$

where $s$ is the reported pooled variance. The trial-level random variable ${\gamma }_{i}$ was used to model a difference in the precision for trial $i$ using a spike-and-slab approach [21]:

$$\text{log}\left({\gamma }_{i}\right)=\left[\right(1-{p}_{i})\times 0]+[{p}_{i}\times {ϵ}_{i}],$$

$${p}_{i}\sim \text{Bernoulli}\left(\theta \right),$$

$$\theta \sim \text{Beta}(1,1),$$

$${ϵ}_{i}\sim \text{N}(0,10).$$

Each trial had a “switch” ${p}_{i}\in (0,1)$, that determined whether it is part of the spike or slab. The spike at zero, with ${p}_{i}=0$, was for trials where the differences between randomised groups are as expected (${\sigma }_{i,j}^{-2}={s}_{i,j}^{-2}$). The slab, with ${p}_{i}=1$, was for trials with under- or over-dispersion. This difference was modelled using a normal distribution where over-dispersed results have a negative ${ϵ}_{i}$ (and multiplier under 1, ${\gamma }_{i}<1$) and under-dispersed results have a positive ${ϵ}_{i}$ (${\gamma }_{i}>1$). The variance for this normal prior of 10 is small compared with typical vague priors in Bayesian models, but in preliminary modelling we found this covered the full range of possibilities, including where the summary statistics were identical between randomised groups, and larger variances had convergence issues. The binary switch for each trial was modelled using a Bernoulli distribution. The hyper-parameter $\theta$ controls is the proportion of trials with potential under- or over-dispersion and had a vague Beta prior, bound between zero and one.

The degrees of freedom (df) for trial $i$ is the total sample size minus one (${N}_{i}-1$), which allows for greater variance in differences for smaller trials.

Two statistics can be used to judge whether a baseline table has under- or over-dispersion:

The estimated trial-specific probability of a non-zero change in the precision ${\overline{p}}_{i}=\sum _{j=1}^{M}{p}_{i}^{\left(j\right)}/M$, for which we examined three thresholds of ${\overline{p}}_{i}>0.50$, $0.80$ and $0.99$ to flag a potential problem by averaging over the $M$ Markov Chain Monte Carlo estimates
The estimated precision ${\overline{ϵ}}_{i}$ which indicates larger under- or over-dispersion for values further from zero. This is also averaged over the Markov Chain Monte Carlo estimates.

We report whether a baseline table has a potential problem, but make no attempt to differentiate between fraud and honest errors [22, 23]. Checking for fraud needs to be done by examining other details, such as ethics clearances, and other work by the same authors. However, our automated check could still be a useful flag when papers are submitted [24].

Our approach uses the baseline table reported by the authors and hence is reliant on reporting accuracy. Inaccurate reporting could lead to issues being flagged incorrectly, for example, authors reporting the standard error of the mean but incorrectly labelling it as the standard deviation. We show below how our model flagged trials where our automated data extraction of the baseline table was unsuccessful or where reporting errors meant the t-statistics were invalid.

Graphical description of the model

A simple graphical description of the model is shown in Fig. 1, which shows hypothetical results for two trials each with 40 comparisons in their baseline table [12].

Both example trials have a mean difference of zero, but the observed differences from one trial are much narrower and closer to zero, indicating that the randomised groups are too similar (under-dispersed). Our approach models the precision (inverse-variance) and aims to detect when trials are under- or over-dispersed using the distribution of all available t-statistics.

Predictors of under- or over-dispersion

We examined whether there were study design characteristics associated with under- or over-dispersion. We used a multiple linear regression model with a dependent variable of the study-level probability of a non-zero dispersion ($0\le {\overline{p}}_{i}\le 1$), and independent variables that described the paper, study design and features of the table. We included independent variables of the journal and country of first author, using a combined “other” if a journal had fewer than 10 trials and a country fewer than 20 trials. For the study design we included if the study was a pilot (based on the title), a cluster-randomised trial (based on the title and abstract), used the standard error of the mean instead of the standard deviation (based on the baseline table), or used block randomisation (based on the full text). The features of the baseline table we included as predictors were the number of rows, number of columns, sample size, largest difference in sample size between groups, proportion of continuous summary statistics and average number of decimal places for summary statistics. We selected a smaller subset of key predictors from the larger set using the elastic net as a variable selection tool [25].

Simulation study

To examine using the t-distribution instead of the distribution of p-values, we simulated data using three examples described by Bland [13] with no concern about randomisation but the p-value distribution would be non-uniform, meaning the uniform test could return a high percentage of false positives:

Small trials with a sample size of 10 and summary statistics using binary data (e.g., percent of males)
Large trials with a sample size of 1,000 and summary statistics using binary data
Large trials with a sample size of 1,000 and summary statistics using skewed continuous data (e.g., length of hospital stay)
To examine the power of our Bayesian method to detect problematic tables, we used three additional scenarios that used a 50:50 mix of binary and continuous data where the summary statistics for the baseline table were:
Under-dispersed, randomised groups were too similar, achieved by copying half of the means and percentages from one group to the other.
Over-dispersed, randomised groups were too different, achieved by adding a large number to the means and percentages.
As expected for a randomised trial.
The split of continuous:binary summary statistics and the parameters for the distributions of group sample size and table rows were based on a large sample of baseline table data from our automated extraction (see Additional file 1). The group sample sizes for these three scenarios were randomly generated using an exponentiated gamma distribution with shape of 10.7 and rate of 2.84, which gives a median sample size of 40 and first to third quartile of 19 to 87. The number of rows per baseline table were randomly generated using a gamma distribution with shape 2.2 and rate 0.15, which gives a median number of rows of 12 and first to third quartile of 7 to 19.

To make realistic tables we rounded the mean to one decimal place and the standard deviation to 2 decimal places. To examine the effect of rounding we repeated the simulation where the dispersion was as expected with a mean rounded to zero decimal places and a standard deviation to 1 decimal place.

For all seven scenarios we created 500 simulated trials and each trial had two randomised groups with equal sample size. We compared the statistics graphically using distributions of the p-values and t-statistics. We tested if the p-values for each study followed a uniform distribution using the Kolmogorov–Smirnov test for uniformity, and counted the number of simulations where the null hypothesis was rejected using the 0.05 threshold. Using our Bayesian model, we examined the number of trials where the estimated probability of the precision was above three thresholds: 0.5, 0.80 and 0.99.

Estimation

All models were estimated using a Bayesian approach using WinBUGS Version 1.4.3 [26] for the paper and nimble version 0.12.1 [27] for the shiny application. The data management and plotting were made using R version 4.1.1 [28]. All the R code to extract the tables are run the model is openly available https://github.com/agbarnett/baseline_tables. An interactive version of our approach is available via shiny: https://aushsi.shinyapps.io/baseline/.

Ethical considerations

We used publicly available data that are published to be read and scrutinised by researchers and hence ethical approval was not required.

Simulation results

Table 1

Results for the simulation study. Percentages of simulated trials that were not flagged or were flagged as under- or over-dispersed (Not flagged / Under-dispersed / Over-dispersed) for three probability thresholds, and a test of uniformity for the p-value distribution.
			Probability threshold
Difference in groups	Simulation type	Uniform test rejected (%)	0.5	0.8	0.99
No	As expected	18.2	96.6 / 2.4 / 1.0	99.4 / 0.2 / 0.4	100.0 / 0 / 0
	Large binary	5.8	98.4 / 1.4 / 0.2	99.6 / 0.4 / 0	100.0 / 0 / 0
	Rounded	23.4	87.4 / 3.8 / 8.8	91.8 / 2.0 / 6.2	96.0 / 1.2 / 2.8
	Skewed	43.6	95.6 / 1.4 / 3.0	98.6 / 1.0 / 0.4	99.0 / 1.0 / 0
	Small binary	31.8	92.2 / 7.8 / 0	97.0 / 3.0 / 0	99.2 / 0.8 / 0
Yes	Over-dispersed	73.6	9.8 / 2.4 / 87.8	13.2 / 1.4 / 85.4	19.0 / 1.2 / 79.8
Yes	Under-dispersed	96.0	26.4 / 70.0 / 3.6	40.4 / 56.0 / 3.6	64.2 / 32.6 / 3.2

The results for the seven simulated scenarios are in Table 1. Five scenarios were generated using the null hypothesis of no difference between groups. The uniform distribution test did badly with high false positive percentages especially when the data were skewed. It did have a good power to detect under- and over-dispersed data. Examples for single simulations are shown in Additional file 1 and show the non-uniform distribution when the data are skewed.

Our algorithm rarely flagged trials where there was no difference in groups, hence there were few false positives. Our algorithm was highly successful at detecting trials that were over-dispersed, with 85.4% of the simulations flagged at the 0.80 threshold. Our algorithm was also reasonably successful at detecting trials that were under-dispersed, with 56.0% of the simulations flagged as under-dispersed at the 0.80 threshold.

Our algorithm had a high accuracy for detecting if p-values were used in the baseline table. For all 72 available trials, the algorithm correctly estimated if a p-value was used or not every time, giving an exact 95% confidence interval for the overall accuracy of 95.0–100%.

Real data results

The majority of the 10,000 potential trials were excluded (Table 2). The three most common reasons were:

the XML file was not available despite being on the open access PubMed Central database,
there was no baseline table or one could not be detected by our algorithm,
it was not a randomised trial.

Table 2

Number of excluded trials and reasons.
reason	n	percent
No baseline table	2,824	37.4%
Full text page not available	2,127	28.2%
Not a randomised trial	1,547	20.5%
No sample size	408	5.4%
Just one column in table	304	4.0%
Follow-up results in baseline table	162	2.1%
Just one sample size	72	1.0%
Pre-post comparison	43	0.6%
Single-arm study	43	0.6%
Difficult layout	14	0.2%
Could not detect statistics	9	0.1%
Total	7,553	100.0%

A previous study found that 92% of trials included a baseline table [1], hence we are likely excluding trials with a baseline table, but one that our algorithm did not detect. Often this was because the baseline table was in graphical format meaning the numbers in the table were not in the XML file. There are likely also exclusions where the study was not a randomised trial, and hence no baseline table was included. There were also trials that did not include a baseline table (e.g., PMC3574512).

The summary statistics extracted from the included trials are shown in Table 3.

Table 3

Summary statistics from the baseline tables and those included in the Bayesian model.
Included	Statistic	n	percent
Yes	Percent	37,291	52
	Continuous	27,544	38
	Numbers	2,818	4
	Confidence interval	59	0
No	Median	4,117	6
	P-values	206	0
	Range	97	0
	Total	72,132	100

There were 2245 included trials with a total of 51,243 table rows. The median number of rows per baseline table was 13 and the median number of columns was 2. The central 50% of published trial dates were between 15 June 2016 and 15 June 2020.

Table 4

Results for the trials from PubMed Central. Number and percentages of trials that were not flagged or were flagged as under- or over-dispersed. Using three thresholds for the study-specific probability of a non-zero change in the precision of 0.50, 0.80 and 0.99.
Probability threshold	No issue	Under-dispersed	Over-dispersed
0.50	66.6	10.6	22.9
0.80	74.6	5.5	20.0
0.99	81.3	2.0	16.6

The results for the trials from PubMed Central are in Table 4 and show that most trials (81.3% for the 0.99 threshold) had no issue. The most common issue was results that were over-dispersed (16.7% for the 0.99 threshold), with fewer trials being under-dispersed (2.0% for the 0.99 threshold).

The t-distributions for three trials that were flagged as over-dispersed are plotted in Fig. 2. For comparison, three randomly selected trials with no issue are also plotted. The three flagged trials were selected using the smallest multiplier of the precision ($ϵ$) and hence show the most extreme over-dispersed trials. In each case there are a small number t-statistics that are extremely large.

Two trials that are flagged are due to errors in our data extraction (PMC7005701 and PMC7301747). For PMC7005701, a categorical variable is wrongly included as continuous, because the variable label included the word “score” which generally meant the summary statistic was continuous. The t-statistic for this row is over 400 and hence the trial is flagged as over-dispersed.

The result for PMC7301747 is an example of where an error in our data extraction creates a false impression of variability. The error occurs due to large numbers such as “15,170 (7,213)” which our algorithm extracts as three statistics: 15170, 7 and 213. This is because a comma is used both for large numbers and as a separator of two statistics such as a range within round brackets. The t-statistic for this row is over 1,000 and hence the trial is flagged as over-dispersed.

The baseline table in trial PMC7302483 had a non-standard layout with four summary statistics per group in four separate columns. The comparisons for this trial were between different summary statistics for the same group.

The t-distributions for three trials that were flagged as under-dispersed are plotted in Fig. 3.

One flagged study was not a randomised trial but was a case–control study with an age and gender matched control group (PMC2176143), hence it was not surprising that the summary statistics in the baseline table were very similar.

One trial used non-standard column headings for the baseline table which meant the data were read in as four rather than two groups (PMC6034465).

One trial labelled proportions ($\in [0,1]$) as percentages ($\in [0,100]$) and hence it looked as if there were lots of zero percentages which meant the two randomised groups appeared highly similar (PMC7578344).

The results from Figs. 2 and 3 show that the most extreme results in terms of precision and variability are often failures in the algorithm’s data extraction. Hence, we next look at less extreme results by excluding flagged trials that are in the tails of the precision distribution, which were the extremely under- or over-dispersed results (see Additional file 1 for the distribution). We show three other examples in Fig. 4 for over-dispersion and in Fig. 5 for under-dispersion.

One trial stratified the randomised groups on a severity variable which created large between group differences and hence the over-dispersion (PMC4074719).

A trial that was flagged as over-dispersed had standard deviations for height that were zero (PMC6230406). This is likely a presentation error as zero standard deviations would require all participants in the two groups to have exactly the same height, with a different common height in each group.

One study was not a trial but was an observational study with some very large differences between groups at baseline, with 4 absolute t-statistics larger than 10 including one that was labelled as not significantly different based on a Mann–Whitney test but had a t-statistic of 19 (PMC6820644).

Using the lower threshold of a multiplier under 10 flags a trial where the t-statistics for all four comparisons are within − 0.3 to 0.3 (PMC3136532). The percentage of women was equal in both groups which the authors said was due to the groups being “blocked by gender,” however, we presume they meant “stratified by gender.”

A trial that was flagged had four t-statistics within − 0.3 to 0.3 (PMC6709840). The randomisation was stratified on age and one of the percentage variables, which somewhat explains the under-dispersion.

A trial that was flagged compared only compared age and gender, but had four groups (PMC7245605). The 12 t-statistics were within − 0.6 to 0.5. The trial was a mix of “healthy controls” and randomised groups, but there was no mention matching for the controls which might explain the under-dispersion.

An alternative method for find under- or over-dispersed trials is to examine where the lower credible interval for the precision multiplier (${ϵ}_{i}$) is above 0. Example under-dispersed trials from this approach a trial with 19 t-statistics between − 0.8 and 0.6 (PMC2885597), and a trial with 8 t-statistics between − 0.8 and 0.6, and with clearly incorrect p-values (PMC6165973).

In the plots above we used histograms to summarise the t-statistics. An alternative approach is using a cumulative distribution function and we show an example of that in Additional file 2.

Predictors of under- or over-dispersion

We examined which study design features were associated with the probability of a non-zero dispersion parameter, indicating under- or over-dispersion. The six predictors selected by the elastic net approach are in Table 5.

Table 5

Estimated predictors of the probability of dispersion showing the mean change in probability and 95% confidence interval.
Predictor	mean	CI
Standard error	0.32	0.21 to 0.43
Difference in group sample sizes of 10+	0.12	0.08 to 0.16
Number of columns (+ 1)	0.10	0.08 to 0.12
Proportion continuous (0.0 to 1.0)	0.10	0.06 to 0.14
Sample size (per doubling)	0.03	0.02 to 0.04
Block randomisation	-0.05	-0.09 to -0.01

The probability of a non-zero dispersion was much higher in baseline tables that wrongly used the standard error of the mean instead of the standard deviation. This is as expected given that the standard error will be far smaller than the standard deviation and hence small differences could look like over-dispersion.

The probability increased when there were large differences in group sample size and for each additional column in the table. In both cases this could be because the trial was not a simple comparison of, for example, treatment versus control (two columns), but might include subgroups, such as gender or disease severity. These strata will likely create over-dispersion as the comparisons are no longer between randomised groups.

The probability increased when the baseline table had a greater proportion of continuous variables. This is likely because of the greater statistical power for continuous variables compared with categorical. Similarly the probability increased with greater sample size and hence greater power. The number of rows in the table was not a predictor, but in a separate simulation we confirmed that—as expected—the power to detect under-dispersion increased for larger tables (see Additional file 1).

Block randomisation was association with a decreased probability of dispersion, and this technique can help balance group characteristics when there is a strong correlation over time in the characteristics of participants. The size of this effect is surprising and it may be that some authors confused block randomisation with stratified randomisation (e.g., PMC3136532) or that these two techniques were often used in combination. We did not add stratified randomisation as a predictor, because “stratified” was often also used to describe analyses, although we could have assumed that any stratification mentioned in the analysis also meant the randomisation was stratified.

No journals or countries were selected by the elastic net variable selection, meaning none were associated with dispersion. However, the total number of trials were small for most journals and some countries, which reduces the statistical power. The largest number of trials for a single journal was 85.

Incorrect use of p-values

Many baseline tables included p-values to compare groups, despite repeated warnings against this [29]. The percentage including p-values in the table was 34% (95% CI 33–36%). With a much larger percentage including p-values in the table or in text referring to the table 64% (95% CI 63–66%). A typical sentence from the text was: “Demographics and disease characteristics were generally balanced between the two arms, except for mean age (Pexa-Vec, 60; BSC, 55 years; p = 0.045; Table 1)” (PMC6682346).

It may be worthwhile to compare groups at baseline using statistical tests for cluster randomised trials [30]. Excluding the 58 cluster randomised trials, the percent including p-values in the table remained high with a mean 35% (95% CI 33–37%).

Our automated algorithm was able to flag baseline tables that would be worth raising with the authors during journal peer review. However, this was not always due to under- or over-dispersion but was sometimes because of an error in the table or because the authors had mislabelled their study as a trial. Flagging these issues with authors at the submission stage could reduce errors and improve reporting. Arithmetic and calculation errors were considered an important and common mistake by medical journal editors [31].

Our algorithm did flag trials that appeared to be under-dispersed. Flagging trials where the baseline table is under-dispersed might protect journals from publishing fraudulent papers, as this has been a clue in previous fraud investigations [22]. It is better to prevent the publication of fraudulent papers, as post-publication retractions which can be long and costly [6]. A study of randomised trials submitted to the journal Anaesthesia, estimated that around one quarter of trials had false data that was problematic enough to invalidate the trial [6]. Research fraud may be increasing due to fiercer competition for funding and promotion that often depend on publication counts [32].

Some of the trials flagged by the algorithm were false positives as the algorithm was unable to correctly read in the baseline table. Often this was because the authors chose a non-standard presentation, and simpler and more consistent baseline tables would improve the accuracy of our algorithm. However, journals are unlikely to prioritise consistent presentation given the many other current issues in peer review [33].

At times our algorithm flagged papers where the baseline table was not a baseline table for a randomised trial, but was a study that re-used the data from a trial. For example, a study which examined responders and non-responders to a randomised treatment, and the table that we extracted which included demographics compared non-randomised groups and so over-dispersion is likely (PMC7660513). It is challenging to exclude these studies as the abstract and title naturally talk about the randomised trial. Any automated flags raised for papers like this would need to be filtered by a person, or the authors whose study was flagged could simply explain that it was not a randomised trial. A group that have designed an automated procedure to detect fake papers containing “tortured phrases” use a two-stage approach where results that are flagged need to be confirmed by a person [34].

Publishers have trialled automated tools to check statistics and reporting completeness [35–37]. If applied by a publisher, our algorithm could be adapted to suit the publisher’s style; the current algorithm tried to cover all journals. The parameters that control which papers are flagged could be tuned with experience to reduce false positives, which are the threshold probability that the precision is non-zero and the size of the precision multiplier.

Using t-statistics

In the simulation study, our model based on the distribution of t-statistics clearly outperformed using the distribution of p-values. Our model dealt well with skewed data and with small samples of binary data, whereas the p-value approach often wrongly flagged these simulated trials as potentially non-random.

Our model did not perform as well when summary statistics for continuous variables used too few decimal places (decimal places do not matter for percentages which are based on the integers of the numerator and denominator). Over-rounding of numbers was relatively common in a study of abstracts, with 12% of percentages wrongly rounded down [38]. Authors should follow the guidelines for decimal places to make papers more readable and facilitate checks like ours [39].

A previous simulation study also found that p-values in a baseline table for categorical data can be non-uniform even for trials that were randomised, and hence recommended against including categorical data [15]. Our method using t-statistics overcomes this problem, and as 54% of summary statistics were numbers or percentages (Table 3) this greatly increases the available data. An advantage of including percentage summary statistics is that they do not have the same rounding issues as continuous summary statistics.

We have created a shiny app where researchers can upload the summary statistics for a trial to examine the distribution of t-statistics and get the results from our Bayesian model https://aushsi.shinyapps.io/baseline/. This avoids the errors of automated data extraction but is more labour intensive. This tool could be useful for researchers who are concerned about particular papers.

Relation to previous work

A number of other automated tools have been used to detect numerical problems in papers, including statcheck for p-values [40] and SPRITE for summary statistics [41]. An automated check of p-values and confidence intervals found up to 5% had significant errors, suggesting there are likely tens of thousands of published papers containing significant but undetected errors [42]. These automated checks had similar motivations to ours: to automate the laborious process of checking numerical results and improve the quality of published papers and/or correct errors in published papers.

We used t-statistics to test for issues in baseline tables, but other methods could be applied such as Benford’s law [22, 43].

Previous studies have examined the percentage of randomised trials that erroneously used p-values to compare randomised groups [1, 44]. A study of over 300 trials from 2010 to 2017 in one journal and found 27% used p-values in tables [44]. An older study examined 50 trials from four high profile journals and found that 48% included significance tests [1]. These are less than our figure of 64% which could be because we include more journals and because we also searched for mentions of “statistical significance” not just p-values. One advantage of so many p-values being presented is that they could be compared with the p-values calculated from the t-statistics and large differences flagged to reduce reporting errors. We found multiple examples where the reported p-values were incorrect.

Individual-level data

Problems can be more accurately detected in individual data than summary statistics and this also avoids any rounding errors [6, 7]. Statistical approaches have been used to detect fraud in trials using individual patient data [4, 7, 45]. Our methods could be extended to examine the dispersion in individual data at baseline, and would likely be far more accurate [6]. Journals could request that authors provide the underlying trial data at submission to perform detailed checks [46, 47]. Authors may raise concerns about participant confidentiality and data security [48], but many data sets in health collect anonymised data and authors need not commit to openly sharing their data or sharing any variables that include personal information.

Improved reporting

There were many potential trials that were excluded because they were not randomised trials (Table 2). Two key reasons for this were poor reporting in the title and abstract [49], and studies that re-used data from a trial in other study designs (e.g., PMC6761647). Some baseline tables were also excluded because of non-standard wording in the caption or because of complex formatting in the table. There is a great variance across journals in how baseline tables are reported, including varied uses of symbols, labels and punctuation marks. We found many mistakes in baseline tables, including misreported statistics (e.g., continuous summary statistics labelled as percentages), missing labels, typographical errors, means reported without standard deviations, zero standard deviations, incorrect confidence intervals, incorrect p-values, and percentages that did not match the numerator divided by the denominator. Researchers should take more care and accurately report their results [50].

Greater use of standardised reporting—such as recommended by CONSORT—would increase the amount and accuracy of data that can be captured using automation. The poor level of compliance with CONSORT is highlighted by the number of trials using p-values to compare randomised groups, a practice that the CONSORT statement warns against, as have multiple statisticians for decades [1]. Even supposedly simple statistics such as age and gender were inconsistently presented in baseline tables, and a previous study similarly found highly varied reporting in age and gender in the clinicaltrials.gov database [19]. Journals that wanted to use our algorithm to screen trials may need to provide more guidance to authors, although journal editors have raised concerns that authors rarely read instructions [51], and there is no systematic study of whether journal instructions are read [52].

Limitations

We categorised papers as being under- or over-dispersed which is arbitrary and does not account for the strength of the dispersion (${ϵ}_{i}$ in the equations). However, when dealing with thousands of papers, some automated decision making is needed. Researchers working with individual papers could focus on the continuous statistic ${ϵ}_{i}$.

We assumed one sample size per group, but there were tables where the sample size varied by row (e.g., PMC7086156) and hence our calculated t-statistics will be inaccurate. Some trials had multiple baseline tables (e.g., PMC7908111), however we just used the first table. Some baseline tables were in an appendix and we only extracted tables from the main text in XML format.

Our simulation examined seven different scenarios, but other likely scenarios could be examined. We assumed a that 50% of statistics were copied when the data were under-dispersed, but fraudsters may copy fewer statistics on average.

Our tool is potentially useful as an initial screening tool of randomised trials, but currently needs human validation of the trials that are flagged as the automated data extraction is imperfect. Similar tools are likely to become more widely used as journals struggle to find reviewers due to increasing submission numbers and over-burdened reviewers [53].

Acknowledgements

Thanks to John Carlisle for useful comments on the study design and first draft.

Competing interests

The authors have declared that no competing interests exist.

Funding

This work was supported by National Health and Medical Research Council (https://www.nhmrc.gov.au/) grant number APP1117784. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Availability of data and materials

All the code to extract the data (which are publicly available) and run the analyses are freely available online: https://github.com/agbarnett/baseline_tables.

Pocock SJ, Assmann SE, Enos LE, et al. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: Current practice and problems. Statistics in Medicine 2002;21:2917–30. doi:10.1002/sim.1296
Schulz KF, Altman DG, and DM. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332–2. doi:10.1136/bmj.c332
Altman DG. Comparability of randomised groups. The Statistician 1985;34:125. doi:10.2307/2987510
Simonsohn U. Just post it. Psychological Science 2013;24:1875–88. doi:10.1177/0956797613480366
Adam D. How a data detective exposed suspicious medical trials. Nature 2019;571:462–4. doi:10.1038/d41586-019-02241-z
Carlisle JB. False individual patient data and zombie randomised controlled trials submitted to Anaesthesia. Anaesthesia 2020;76:472–9. doi:10.1111/anae.15263
Buyse M, George SL, Evans S, et al. The role of biostatistics in the prevention, detection and treatment of fraud in clinical trials. Statistics in Medicine 1999;18:3435–51. doi:10.1002/(sici)1097-0258(19991230)18:24<3435::aid-sim365>3.0.co;2-o
Li W, Wely M van, Gurrin L, et al. Integrity of randomized controlled trials: Challenges and solutions. Fertility and Sterility 2020;113:1113–9. doi:10.1016/j.fertnstert.2020.04.018
Roberts I, Smith R, Evans S. Doubts over head injury studies. BMJ 2007;334:392–4. doi:10.1136/bmj.39118.480023.be
Bolland MJ, Avenell A, Gamble GD, et al. Systematic review and statistical analysis of the integrity of 33 randomized controlled trials. Neurology 2016;87:2391–402. doi:10.1212/wnl.0000000000003387
Bordewijk EM, Li W, van Eekelen R, et al. Methods to assess research misconduct in health-related research: A scoping review. Journal of Clinical Epidemiology 2021;136:189–202. doi:https://doi.org/10.1016/j.jclinepi.2021.05.012
Carlisle JB, Dexter F, Pandit JJ, et al. Calculating the probability of random sampling for continuous variables in submitted or published randomised controlled trials. Anaesthesia 2015;70:848–58. doi:10.1111/anae.13126
Bland M. Do baseline p-values follow a uniform distribution in randomised trials? PLOS ONE 2013;8:1–5. doi:10.1371/journal.pone.0076010
Bolland MJ, Gamble GD, Avenell A, et al. Rounding, but not randomization method, non-normality, or correlation, affected baseline p-value distributions in randomized trials. Journal of Clinical Epidemiology 2019;110:50–62. doi:10.1016/j.jclinepi.2019.03.001
Bolland MJ, Gamble GD, Avenell A, et al. Baseline p value distributions in randomized trials were uniform for continuous but not categorical variables. Journal of Clinical Epidemiology 2019;112:67–76. doi:10.1016/j.jclinepi.2019.05.006
Kennedy ADM, Torgerson DJ, Campbell MK, et al. Subversion of allocation concealment in a randomised controlled trial: A historical case study. Trials 2017;18. doi:10.1186/s13063-017-1946-z
Weissgerber T, Riedel N, Kilicoglu H, et al. Automated screening of COVID-19 preprints: Can we help authors to improve transparency and reproducibility? Nature Medicine 2021;27:6–7. doi:10.1038/s41591-020-01203-7
Marshall IJ, Nye B, Kuiper J, et al. Trialstreamer: A living, automatically updated database of clinical trial reports. Journal of the American Medical Informatics Association 2020;27:1903–12. doi:10.1093/jamia/ocaa163
Cahan A, Anand V. Second thoughts on the final rule: An analysis of baseline participant characteristics reports on ClinicalTrials.gov. PLOS ONE 2017;12:e0185886. doi:10.1371/journal.pone.0185886
D’agostino RB, Chase W, Belanger A. The appropriateness of some common procedures for testing the equality of two independent binomial populations. The American Statistician 1988;42:198–202. doi:10.1080/00031305.1988.10475563
Ishwaran H, Rao JS. Spike and slab variable selection: Frequentist and Bayesian strategies. The Annals of Statistics 2005;33. doi:10.1214/009053604000001147
George SL, Buyse M. Data fraud in clinical trials. Clinical Investigation 2015;5:161–73. doi:10.4155/cli.14.116
Mascha EJ, Vetter TR, Pittet J-F. An appraisal of the Carlisle-Stouffer-Fisher method for assessing study data integrity and fraud. Anesthesia & Analgesia 2017;125:1381–5. doi:10.1213/ane.0000000000002415
Bero L. Stamp out fake clinical data by working together. Nature 2022;601:167–7. doi:10.1038/d41586-022-00025-6
Hastie T, Tibshirani R, Wainwright M. Statistical learning with sparsity: The lasso and generalizations. CRC Press 2015.
Lunn DJ, Thomas A, Best N, et al. WinBUGS - a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing 2000;10:325–37. doi:10.1023/a:1008929526011
de Valpine P, Paciorek C, Turek D, et al. NIMBLE: MCMC, particle filtering, and programmable hierarchical modeling. 2021. doi:10.5281/zenodo.1211190
R Core Team. R: A language and environment for statistical computing. Vienna, Austria: : R Foundation for Statistical Computing 2021.https://www.R-project.org/
Boer MR de, Waterlander WE, Kuijper LD, et al. Testing for baseline differences in randomized controlled trials: An unhealthy research behavior that is hard to eradicate. International Journal of Behavioral Nutrition and Physical Activity 2015;12. doi:10.1186/s12966-015-0162-z
Bolzern JE, Mitchell A, Torgerson DJ. Baseline testing in cluster randomised controlled trials: Should this be done? BMC Medical Research Methodology 2019;19. doi:10.1186/s12874-019-0750-8
Fernandes-Taylor S, Hyun JK, Reeder RN, et al. Common statistical and research design problems in manuscripts submitted to high-impact medical journals. BMC Research Notes 2011;4. doi:10.1186/1756-0500-4-304
Gopalakrishna G, Riet G ter, Vink G, et al. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in the Netherlands. Published Online First: July 2021. doi:10.31222/osf.io/vk9yt
Tennant JP, Ross-Hellauer T. The limitations to our understanding of peer review. Research Integrity and Peer Review 2020;5. doi:10.1186/s41073-020-00092-1
Cabanac G, Labbé C, Magazinov A. Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals. CoRR 2021; abs/2107.06751.https://arxiv.org/abs/2107.06751
BMJOpen. BMJ Open trials Penelope. 2017.https://blogs.bmj.com/bmjopen/2017/02/06/bmj-open-trials-penelope/
Frontiers. AI-enhanced peer review: Frontiers launches next generation of efficient, high-quality peer review. 2018.https://blog.frontiersin.org/2018/12/14/artificial-intelligence-peer-review-assistant-aira/
Heaven D. AI peer reviewers unleashed to ease publishing grind. Nature 2018;563:609–10. doi:10.1038/d41586-018-07245-9
Barnett AG. Missing the point: Are journals using the ideal number of decimal places? F1000Research 2018;7:450. doi:10.12688/f1000research.14488.3
Cole TJ. Too many digits: The presentation of numerical data. Archives of Disease in Childhood 2015;100:608–9. doi:10.1136/archdischild-2014-307149
Nuijten MB, Polanin JR. “statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses. Research Synthesis Methods Published Online First: 2020. doi:10.1002/jrsm.1408
Heathers JA, Anaya J, Zee T van der, et al. Recovering data from summary statistics: Sample parameter reconstruction via iterative TEchniques (SPRITE). Published Online First: May 2018. doi:10.7287/peerj.preprints.26968v1
Wren JD. Algorithmically outsourcing the detection of statistical errors and other problems. The EMBO Journal 2018;37. doi:10.15252/embj.201899651
Bradshaw MS, Payne SH. Detecting fabrication in large-scale molecular omics data. PLOS ONE 2021;16:1–5. doi:10.1371/journal.pone.0260395
Ahn EJ, Kim JH, Kim TK, et al. Assessment of p values for demographic data in randomized controlled trials. Korean Journal of Anesthesiology 2019;72:130–4. doi:10.4097/kja.d.18.00333
Wu X, Carlsson M. Detecting data fabrication in clinical trials from cluster analysis perspective. Pharmaceutical Statistics 2010;10:257–64. doi:10.1002/pst.462
Roberts I, Ker K, Edwards P, et al. The knowledge system underpinning healthcare is not fit for purpose and must change. BMJ 2015;350:h2463–3. doi:10.1136/bmj.h2463
Shamseer L, Roberts J. Disclosure of data and statistical commands should accompany completely reported studies. Journal of Clinical Epidemiology 2016;70:272–4. doi:10.1016/j.jclinepi.2015.05.033
Hardwicke TE, Ioannidis JPA. Populating the data ark: An attempt to retrieve, preserve, and liberate data from the most highly-cited psychology and psychiatry articles. PLOS ONE 2018;13:e0201856. doi:10.1371/journal.pone.0201856
Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. The Lancet 2014;383:267–76. doi:10.1016/s0140-6736(13)62228-x
Prager EM, Chambers KE, Plotkin JL, et al. Improving transparency and scientific rigor in academic publishing. Journal of Neuroscience Research 2018;97:377–90. doi:10.1002/jnr.24340
Tobin MJ. Authors, authors, authors follow instructions or expect delay. American Journal of Respiratory and Critical Care Medicine 2000;162:1193–4. doi:10.1164/ajrccm.162.4.ed-9-00
Malički M, Jerončić A, Aalbersberg IjJ, et al. Systematic review and meta-analyses of studies analysing instructions to authors from 1987 to 2017. Nature Communications 2021;12. doi:10.1038/s41467-021-26027-y
Severin A, Chataway J. Overburdening of peer reviewers: A multi-stakeholder perspective on causes and effects. Learned Publishing 2021;34:537–46. doi:10.1002/leap.1392

No competing interests reported.

fraudpapersupplement.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Status:

Version 1

Abstract

Figures

Introduction

Methods

Automated extraction of baseline tables

Validation

Creating t-statistics for group differences

Bayesian model of observed differences

Graphical description of the model

Predictors of under- or over-dispersion

Simulation study

Estimation

Ethical considerations

Results

Simulation results

Real data results

Predictors of under- or over-dispersion

Incorrect use of p-values

Discussion

Using t-statistics

Relation to previous work

Individual-level data

Improved reporting

Limitations

Conclusion

Declarations

Acknowledgements

Competing interests

Funding

Availability of data and materials

References

Additional Declarations

Supplementary Files

Status:

Version 1