Baseline categorical variables in spine randomized controlled trials

doi:10.21203/rs.3.rs-3622260/v1

Download PDF

Research Article

Baseline categorical variables in spine randomized controlled trials

https://doi.org/10.21203/rs.3.rs-3622260/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Recently Levayer and colleagues surveyed randomized controlled trials in 4 spine journals for signs of integrity issues based on an assessment of baseline p-values for categorical data.

They concluded that there was no evidence of “systemic fraudulent behaviour”, and that data were consistent with random allocation.

Methods

We used their published dataset to compare reported with independently calculated baseline p-values and the observed and expected distributions of frequency counts and baseline p-values using the reappraised package for R.

Results

In 51/929 (5.5%) baseline variables, the sum of frequencies by level did not agree with the reported number of participants. For one third of reported p-values (172/522), we could not calculate a matching p-value using a range of standard statistical tests. Sparse data were common; for 22% (74/332) variables in which the reported p-value matched a calculated p-value from a chi-square test, the expected cells were smaller than are recommended for use of chi-square tests. There were 20-25% more two-arm trials with between-groups differences in frequency counts of 1 or 2 than expected. There were small differences between the observed and expected distributions of baseline p-values, but these were dependent on how sparse data were analysed.

Conclusion

Incorrectly reported p-values and incorrect usage of statistical tests are common in these spine journals. There are differences between observed and expected distributions of baseline p-values and frequency counts. Collectively, the findings do raise questions about the reliability of some RCTs in major spine journals.

Anesthesiology & Pain Medicine

spine

randomised controlled trials

p-values

randomisation

statistical tests

Levayer and colleagues surveyed 167 randomized controlled trials (RCTs) in 4 selected “spine” journals for evidence of non-random baseline frequency data, concluding that there was no widespread evidence of data fabrication or non-random allocation [1]. The authors should be congratulated for such a major undertaking and making data available with their publication. To our knowledge, this is the only published dataset of baseline categorical variables from a large group of RCTs.

We have previously reported that baseline categorical data can be used to assess publication integrity of groups of RCTs [2–4], and have published a freely available package ‘reappraised’ for the R statistical software program (https://CRAN.R-project.org/package=reappraised). We took the opportunity to assess the dataset created by Levayer and colleagues [1] using the tools in the reappraised package.

We imported the published excel spreadsheets [1] into R and assembled them into a single data frame. This generated a dataset with 167 studies and 940 variables, of which 11 variables were ignored or excluded by Levayer and colleagues, leaving 929 variables for analysis- the “spine dataset”. In the original published dataset there are 924 variables in the summary table (column ‘nb of bin variables’), but Levayer and colleagues state they used 921 in their analysis [1]. The difference of 5 variables between the spine dataset we created and the summary table was: 1 variable with a reported and calculated p-value was not analysed; 1 variable with a reported p-value but no calculated p-value (and a large difference between the sum of frequencies and the number of participants) was not analysed; and 3 variables with only one level and with no reported or calculated p-values and either 0 or 100% frequency for the solitary level were not analysed. We included all 5 variables in the spine dataset.

Chi-square tests are not appropriate in the presence of sparse data. We used the commonly applied rule of at least one cell with an expected value < 5 for 2 by 2 tables, and at least one cell with an expected value of < 1 or > 20% of cells with an expected value of < 5 for tables larger than 2 by 2 as the thresholds for indicating sparse data such that the expected cells are too small for the chi-square test and that an exact test should be used. When Fisher’s exact test is performed on tables larger than 2 by 2, the computational memory required can become very large with long running times when the tables are large. We used 10,000 Monte-Carlo simulations in the R fisher.test function to avoid this issue.

Baseline variables:

For 51/929 (5.5%) variables, the sum of frequencies by level for each variable did not agree with the reported total number of participants. For 11 (1.2%) variables, the sum of frequencies exceeded the total number of participants, implying data extraction errors by Levayer and colleagues, typographical errors by either Levayer and colleagues or the trial authors, or impossible data. For example, one study had 9 smokers and 38 non-smokers (47 total) but only 43 participants. 5 of the 11 differences were 1, another 3 were < 10, and 3 were > 10 (percent difference range 0.4–32%). For a further 40 (4.3%) variables, the sum of frequencies for each variable was smaller than the total number of participants, again implying data extraction errors, typographical errors, missing data, or other errors. 31/167 (19%) of studies had variables with either of these issues, and 12 (7%) studies had both.

Comparing reported and calculated baseline p-values

Levayer and colleagues extracted p-values from the original publications for the comparison between groups for 495/871 baseline variables (‘baseline p-values’) in 166 studies for which the sum of the frequencies equalled the number of participants and there was at least one frequency count > 0 (the reported p-values). The statistical test used to generate the p-values in the original publications was not described in the spine dataset. For these 495 variables, 481 had a numeric p-value. We independently calculated the p-value for each variable from the summary statistics in the spine dataset and rounded it to the same number of decimal places as the reported p-value. Table 1 shows that 328 (68%) reported p-values matched with a calculated p-value (217 chi-square test, 3 chi-square test with continuity correction, 100 Fisher’s exact test, 4 mid-p test, and 4 other tests). Of the 153 variables (32%) with mismatched reported and calculated p-values, the difference between the reported p-value and the closest calculated p-value ranged from 0.0001 to 1. Of the 14 variables with a reported threshold p-value (p > 0.05, > 0.99, > 0.999, or > 0.9999), 11/14 (79%) had a matching calculated p-value. The 3 non-matching values were: reported p-value > 0.9999; closest calculated p-value 0.69, 0.86, and 0.86 respectively. Overall, mismatched reported and calculated p-values were present in 62/166 (37%) studies.

Table 1

Difference between reported and calculated baseline p-values
		Baseline p-value
Difference between reported and calculated p-values	n (%)	0–0.1	0.1–0.2	0.2–0.3	0.3–0.4	0.4–0.5	0.5–0.6	0.6–0.7	0.7–0.8	0.8–0.9	0.9–1
0	328 (68)	31	22	31	26	43	31	37	28	34	45
0–0.1	96 (20)	9	5	11	5	14	11	9	10	8	14
0.1–0.2	29 (6)		1	2	4	4	9	1	4	1	3
0.2–0.3	8 (2)		1			2	3		2
0.3–0.4	8 (2)		3			1	1	3
0.4–0.5	5 (1)			1	2	1				1
0.5–0.6	2 (0)			1				1
0.6–0.7	4 (1)	1	1					1	1
0.9–1	1 (0)										1

Of the 220 variables with a matching calculated p-value from a chi-square test, 44 (20%, 20 from 2 by 2 tables, 24 from tables larger than 2 by 2) had expected cells that were too small for the chi-square test. Overall, 74/332 (22%) chi-square tests in 51/167 (30%) studies were done even though the data were sparse.

We repeated the analyses for the 51 variables in which the frequency totals did not agree with the total participants, using the reported frequency counts. 25 (45%) had a reported p-value and all were numeric. 10 (40%) of the reported p-values had a matching calculated p-value (8 chi-square test, 2 Fisher’s exact). The differences for non-matching p-values ranged from 0.002 to 1. 8/17 (47%) chi-square tests had expected cells that were too small (6 tables larger than 2 by 2, and two 2 by 2 tables), in 7 different studies.

Finally, there were 7 variables in which all the frequency counts were either 0 or the total number of participants in each group. For these variables, a test on a 2 by 2 table cannot be performed. However, a one-way chi-square test can be done. Two variables had a reported p-value (both 1). Only 1 of the reported p-values matched that obtained from a one-way chi-square test (1, and 0.06 respectively).

In the entire spine dataset, there were 522 variables with reported p-values. For 172 (33%) of these reported p-values, we could not calculate a matching p-value using a range of standard statistical tests. 82/522 variables (16%) had expected cells that were too small for the chi-square test, and of these, 47 had a reported p-value that matched with a p-value calculated with a chi-square test.

Distribution of frequency counts:

We assessed the differences between groups for frequency counts in 135 two-arm trials with 677 variables for which the sum of the frequency counts matched the total participants. We randomly excluded a single level to avoid double counting. Figure 1 shows that there was an excess of variables with a difference of 1 or 2 compared to the expected distribution. The differences between the observed and expected differences of 1 and 2 were 29 and 18 respectively. When we included variables in which the total frequency counts differed from the total participants, there were 138 trials, 713 variables, and the results were similar.

Distribution of baseline p-values:

Finally, we assessed the distribution of calculated baseline p-values compared with the empirically calculated distribution. We performed analyses separately on variables where the sum of the frequency counts agreed with the total number of participants and variables where they did not agree. We present results for variables where they agreed, although the results were similar for the analyses of all variables. The results were heavily influenced by the choice of statistical test because many of the variables contained sparse data where the expected cells were too small for the chi-square test (294/922 (32%) variables in spine dataset without frequency count of 0 or 100%, 134/642 (21%) 2 by 2 tables; 160/280 (57%) larger than 2 by 2 tables). Levayer and colleagues used chi-square tests for all variables in their primary analyses [1]. However, they noted the issues with sparse data and used mid-p tests to analyse one small study in a secondary analysis [1]. For tables with more than 2 rows or columns, Fisher’s exact test is the only easily accessible option for sparse data.

Figures 2A and 2B show the distribution of p-values in variables which had 2 treatment groups and 2 levels (ie 2 by 2 table) with sparse data. They show that the distribution of p-values, both observed and expected, differ markedly depending on whether mid-p or Fisher’s exact test are used. The observed distribution differed from the expected distribution for the mid-p test but was similar for the Fisher’s exact test. Figure 2C and 2D show that the observed and expected distributions were similar for p-values from variables without sparse data when calculated with the chi-square test or the combination of mid-p test for 2 by 2 tables and chi-square test for larger tables. Figures 2E and 2F show the distribution of p-values in all variables using either chi-square tests for variables without sparse data and Fisher’s exact test for variables with sparse data, or mid-p tests for 2 by 2 tables, and chi-square (without sparse data) or Fisher’s exact test (with sparse data) for larger tables. For both analyses, the observed and expected distributions were visually fairly similar but there were statistically significant differences between them.

The issue regarding statistical tests is illustrated well by an example from the paper by Levayer and colleagues [1]. A trial with 30 participants reported 11 variables and baseline p-values were calculated using the chi-square test to give a study-wise p-value of 0.996. 7/11 variables had expected cells < 5. The baseline p-values were recalculated using a mid-p test giving a study-wise p-value of 0.92, which the authors said was reassuring. However, the actual distribution of mid p-values was unusual- 7/11 mid-p values were between 0.6–0.8, and 10/11 between 0.49 and 0.85. The expected distribution calculated empirically was a “noisy approximately uniform” distribution as shown in Fig. 3, and the observed distribution differed markedly from this.

Implications for the reliability of trials in spine journals:

In these analyses, we were unable to independently replicate the p-value calculation for about one third of variables with a reported baseline p-value that had been extracted by Levayer and colleagues. Of those we could replicate, about 1 in 5 used a chi-square test even though there were sparse data with expected cells too small for the chi-square test, indicating that an exact test (eg mid-p or Fisher’s exact test) should be considered. In two-arm trials, there was a 20–25% excess of variables in which the between-groups difference in frequency counts was 1 or 2 compared to the expected amount. Similar findings have been reported in groups of trials with integrity concerns [4]. There were small differences between the observed and empirically calculated distributions of baseline p-values, particularly for those with sparse data. However, the importance of these findings remains uncertain. Collectively these issues raise concerns about the reliability of some of the trials in this dataset.

Comparison to the previous results

Levayer and colleagues concluded that there was no evidence of systemic fraudulent behaviour, or non-random allocation in these trials [1]. They commented that their approach “will detect fraud, provided it presents as non-random allocation.” They therefore suggested that readers can “generally assume that RCTs published in these spine journals are genuine” [1].

We think that any suggestions of fraud based on assessment of publications should be avoided, because such assessments can only determine the existence of issues such as errors, impossible or improbable data, plagiarism, image manipulation, and statistical patterns inconsistent with randomisation [5]. These issues can be grouped under the term publication integrity, which can be readily assessed. However, if compromised integrity is identified, the reasons for the problems (such as honest error or questionable research practices including fraud or fabrication) can only be determined from an investigation [5]. The reason for the compromised integrity matters little to the readers of the publications, whose main concern is that the publication is reliable. For these reasons, we think it unwise to make any inferences about the presence or absence of fraud in these trials.

Putting that important issue aside, there are a number of other reasons to be cautious. Firstly, 167 trials were assessed, but 11 had only a single baseline p-value, and 98 (59%) had 5 or fewer p-values. The number of p-values in a study needed to draw a reliable conclusion from analyses of baseline p-values is not known, but it is likely to be considerably higher than 5, unless the p-values are extremely skewed. 3/69 (4%) and 2/10 (20%) trials with at least 5 or 10 baseline p-values, respectively, were identified using the threshold of a study-wise p-value of > 0.99, and a further 5 trials with > 5 baseline p-values using the thresholds of < 0.05 or > 0.95. One of the 3 trials exceeding the 0.99 threshold has been retracted. This seems less reassuring than Levayer and colleagues’ conclusion.

Furthermore, the test the authors used, the Carlisle-Fisher-Stouffer method only detects situations where there is an excess of baseline p-values close to 0 or close to 1. Carlisle made it clear that he was looking for outliers: trials that did not conform to this test of baseline data [6]. Carlisle was not seeking to detect every case of fraud [6]. Outlying results might have many explanations, one of which, and perhaps the least likely, is fabricated data. An example where this technique does not work well is that a study could have 10 baseline p-values between 0.45 and 0.55, a very unlikely distribution, but the study-wise p-value would be close to 0.5. In addition, Carlisle used the technique for continuous data, and to our knowledge, it has not been applied to categorical data previously. Categorical data differ from continuous data in that the baseline p-values are not uniform, and, particularly when the sample size is small, there are only a small number of discrete p-values [2]. The combination of small sample sizes and sparse data means a high proportion of p-values calculated using Fisher’s exact test will be 1 (Fig. 2A). However, Levayer and colleagues, chose to use the chi-square rather than exact tests which may have impacted the study-wise p-values calculated. If there is a baseline p-value of 1, the z-score used in the Carlisle-Fisher-Stouffer method cannot be derived. Levayer and colleagues chose to substitute a value of 3, comparable to a p-value of 0.998, including for the situation where all the frequency counts were 0 (eg all or none of the participants in the trial had the characteristic). It is not certain what the best approach should be for these situations- possibilities could include excluding sparse data, calculating one-way p-values for variables with frequencies counts of 0, or using different statistical tests. However, if different statistical tests are used, the baseline p-value distribution may change substantially but remain different from the expected distribution (Figs. 2 and 3).

Limitations

We relied on the publicly available dataset and did not compare these data with the original publications, and so it is possible that some of the issues identified might be due to data extraction or typographical errors. The dataset did not contain the statistical test used to generate or report the p-values in the original trials. However, the reappraised package function calculates p-values using 7 commonly used statistical tests, allowing comparisons to results from all the major statistical approaches. While we have identified some issues about the publication integrity of this broad group of trials, we cannot make comments about individual RCTs since we have not assessed any individual trial in detail, and because some of the assessments (p-value distribution and frequency distribution) were based on the whole group of trials, preventing inferences about individual RCTs. Finally, we only assessed categorical data in the spine dataset. Continuous variables provide a number of options for analysis, which may be more preferable to analysing categorical variables [2].

Summary

The dataset published by Levayer and colleagues has allowed a more detailed analysis of baseline categorical data in 167 RCTs in spine journals. This showed that incorrectly reported p-values and incorrect usage of statistical tests are common, and that there are differences between the observed and expected distributions of both frequency counts of variables and baseline p-values. Collectively, these findings raise questions about the reliability of some spine RCTs.

A simple potential fix for these issues would be for journals to require authors to provide a table at submission that contains baseline summary continuous and categorical data and p-values for the trial in a standardized format that would permit automated extraction and examination of data. If problems are identified, explanations could be sought during the review process for incorrect baseline p-values or unusual distributions or matching of baseline data. Publishing this table as supplementary information might be useful because, for example, reporting of baseline p-values is considered unnecessary by many experts [7]. Journals should also ensure that the statistical tests applied to categorical variables conform to recommended practice. Independent analysis and publication of anonymized individual patient data is likely to be much more informative for assessment of publication integrity [8].

Competing Interests: We have no financial conflicts to declare. MB is the author of the reappraised statistical package.

Funding: This research received no specific funding. MB is a recipient of a Health Research Council (HRC) of New Zealand Clinical Practitioners Fellowship. The Health Services Research Unit is funded by the Chief Scientist Ofﬁce of the Scottish Government Health and Social Care Directorates. The authors are independent of the HRC. The HRC had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

MB, AA, and AG all were involved in the study design. MB did the analysis and drafted the paper. All authors critically reviewed and improved it. All authors read and approved the final manuscript

Levayer MMS, Chew GRP, Sheldrick KA, Diwan AD. Characteristics of baseline frequency data in spinal RCTs do not suggest widespread non-random allocation. Eur Spine J. 2023:Published online June 12.
Bolland MJ, Gamble GD, Avenell A, Grey A, Lumley T. Baseline P value distributions in randomized trials were uniform for continuous but not categorical variables. J Clin Epidemiol. 2019;112:67-76.
Bolland MJ, Gamble GD, Avenell A, Cooper DJ, Grey A. Participant withdrawals were unusually distributed in randomized trials with integrity concerns: a statistical investigation. J Clin Epidemiol. 2021;131:22-29.
Bolland MJ, Gamble GD, Avenell A, Cooper DJ, Grey A. Distributions of baseline categorical variables were different from the expected distributions in randomized trials with integrity concerns. J Clin Epidemiol. 2023;154:117-124.
Grey A, Bolland MJ, Avenell A, Klein AA, Gunsalus CK. Check for publication integrity before misconduct. Nature. 2020;577:167-169.
Carlisle JB. Data fabrication and other reasons for non-random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals. Anaesthesia. 2017;72:944-952.
Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869.
Carlisle JB. False individual patient data and zombie randomised controlled trials submitted to Anaesthesia. Anaesthesia. 2021;76:472-479.

Download PDF

Version 1

posted

You are reading this latest preprint version

Baseline categorical variables in spine randomized controlled trials

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Baseline variables:

Comparing reported and calculated baseline p-values

Distribution of frequency counts:

Distribution of baseline p-values:

Discussion

Implications for the reliability of trials in spine journals:

Comparison to the previous results

Limitations

Summary

Declarations

References

Status:

Version 1