Assessing the Quality of Randomized Controlled Trials of Complex Regional Pain Syndrome Published in the Journal of Clinical Pain Medicine Field

Background: Complex regional pain syndrome is a disease characterized by chronic pain caused by tissue damage and worsens over time. The quality of randomized controlled trials on complex regional pain syndrome is low due to the rarity of the disease and diculty in designing studies. Therefore, we evaluated the completeness of evidence by assessing the quality. Methods: We searched articles that were published between 1998 and 2017. The quality was assessed using the Jadad scale, van Tulder scale, and the Cochrane collaboration risk of bias tool. Results: A total of 72 articles on complex regional pain syndrome have been published. Only 31 articles were found to have a low risk of bias according to the Cochrane Collaboration. On the Jadad and van Tulder scale, 52 and 58 articles, respectively, were assessed as high quality. Only a few trials described the randomization method adequately and presented allocation concealment correctly. We also found that research trials with funding statements had higher Jadad scales than unfunded (P = 0.03). Furthermore, articles that described the sample size were of signicantly higher quality (P < 0.01) than those that did not. Conclusions: There was no observed improvement in quality over time. In particular, the performance rate of the allocation concealment for the analyzed papers was low. In addition, we found that the number of high-quality articles increased when institutional review board approval was granted and funding provided. For the future, we recommend that researchers focus their efforts on conducting high-quality studies.


Background
Evidence-based Medicine (EBM) refers to the rational approach to individual patient care with an emphasis on integrating data from clinical studies on diagnosis, treatment, and disease prevention (1,2).Increasingly, EBM is based on the randomized controlled trial (RCT) approach (3).The typical design of RCTs minimizes the risk of bias, optimizes the level of evidence, and ensures the generation of highly reliable results (4).However, RCT design does not eliminate bias completely.Bias that arises at the RCT design, performance, reporting, or application phases may lead to an underestimation or overestimation of the true intervention effect (5).Since RCT results are applied in clinical practice, RCTs should involve high standards in order to minimize bias (6).
An objective methodological quality evaluation of RCTs is certainly a necessary review process to enable scienti c medical treatment (7).At the time of writing, the International Committee of Medical Journal Editors recommends the Consolidated Standards of Reporting Trials (CONSORT) statement as a guideline for RCT quality improvement (8).The CONSORT guidelines assist physicians in the understanding, performance, and analysis of RCTs (9).However, the CONSORT statement is not intended as a tool for the evaluation of RCT quality.
The quality assessment for an RCT report involves evaluation of whether the study design, procedures, and statistical analyses justify the authors' conclusions (10).This process therefore identi es erroneous data and unsubstantiated conclusions that require exclusion from clinical application, and thereby reduces the potential for non-justi ed medical costs (11).Methods to evaluate RCT quality include checklists, scales, and the assessment of individual indices.To evaluate bias, a Jadad quality assessment scale (12) is widely used in the evaluation of RCT quality.However, a further requirement in RCTs is the avoidance of selection bias via appropriate treatment allocation and allocation concealment (13), which are factors that are not typically addressed by the Jadad scale.Tools comprising items for allocation concealment include the van Tulder scale (VTS) (14) and the Cochrane Collaboration Risk of Bias Tool (CCRBT) (15).
Complex regional pain syndrome (CRPS) is associated with various disorders, such as localized skin color and temperature changes, edema, and severe pain, but it is not yet fully understood (16,17).Even after the development of new diagnostic criteria, disagreements about CRPS diagnoses and treatments continue (17).Physicians and patients have a low understanding of CRPS, and thus, the time period before patients are treated by pain specialists is usually long (18).Although several RCTs have been reported on CRPS, the quality of articles is typically lower than it could be due to the rarity of the disease itself and di culty in designing studies (19,20).Therefore, we evaluated the completeness of the presented evidence by assessing the quality of RCTs on CRPS.
Although the evaluation of RCT quality has been the subject of several recent studies (3,7,11,21,22) to date, no study has performed a systematic evaluation of the quality of RCTs on CRPS.

Subjects
To identify all RCT articles on CRPS between 1998 and 2017, two independent authors (Park and Ryu) performed a manual search of PubMed, EMBASE, and Cochrane library.Then, a third researcher (Shin) adjusted the two sets of data in order to select articles for inclusion in the present study.

Assessment of RCT quality
RCT quality was assessed using the Jadad scale, the VTS, and the CCRBT by the two independent researchers.

Jadad Scale
The Jadad scale score comprises a total of ve points: two points for randomization, two points for blinding, and one point for dropout.If the total score is > 3 points, the RCT is considered high quality.If the total score is < 2 points, the RCT is considered low quality.For RCTs which had a precluded doubleblind design, they are considered high quality when the total score is > 2 (12).van

Tulder Scale
The VTS assesses the following 11 aspects of RCT quality: randomization, allocation concealment, baseline characteristics, patient blinding, care provider blinding, observer blinding, co-intervention, compliance, drop-out rate, end-point assessment time point, and intention-to-treat analysis (14).For each item, the response 'yes', 'no', or 'do not know' is applied.If the criteria for ve or more items are met (> 5 points), the RCT is evaluated as high quality (14).

Cochrane Collaboration risk of bias tool
The CCRBT assesses the following six domains of RCT quality: sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity (15).Each domain is assessed using a 'yes', 'no', or 'unclear' response, which indicate a low risk of bias, a high risk of bias, and an uncertain risk of bias, respectively.If the response in the rst three domains is 'yes', and no important concerns are identi ed in relation to the last three domains, the RCT is classi ed as having a low risk of bias.If the response in ≤ 2 domains is 'unclear' or 'no', the study is classi ed as having a moderate risk of bias.If the response in ≥ 3 domains is 'unclear' or 'no', the RCT is classi ed as having a high risk of bias (23).

Evaluation of RCT quality according to other factors
Changes in frequency over time of statements concerning sample size, con ict of interest, funding sources, institutional review board (IRB) approval, double blinding, and allocation concealment were determined.

Statistical analysis
Analysis of RCT quality was conducted at 5-year intervals from 1998.For each assessment tool, the score and time ow were compared using a one-way ANOVA.Chi-square tests were used to analyze the association between high quality article status according to the Jadad score and according to the CCRBT.Fisher`s exact tests were used to analyze the associations between high quality article status according to the Jadad score, VTS, and CCRBT ndings.A Student's t-test was used to analyze IRB approval, blinding, funding, con ict of interest, allocation, sample size, and country distribution.SPSS version 24.0 (SPSS Inc., Chicago, Illinois, USA) was used for statistical analysis and a P-value < 0.05 was considered to be statistically signi cant.

Analysis of topics related to the articles
This is the rst study to assess the quality of RCTs on CRPS in the eld of pain medicine.There was no distinct intervention therapy in the management of CRPS during the period of 1998-2007, but from 2008-2017 pain physicians tried several treatments, such as rehabilitation therapy, drugs, and regional blocks, to reduce the complications of CRPS syndrome (Table 1).With respect to drug therapy, N-Methyl-Daspartate (NMDA) blockers were most commonly investigated (Table 1).

Characteristics of RCTs according to publication year.
In total, 72 original RCT articles on CRPS were published between 1998 and 2017 (Figure 1).The number of RCTs reported in the 1998-2002, 2003-2007, 2008-2012, and 2013-2017 time periods were 8, 14, 25, 25, respectively (Table 2).The complete reports, including sample sizes and con icts of interest, were described in 37 (51.4%) and 6 (8.3%) of the RCTs, respectively (Table 2).The funding source was described in 53 (73.6%) papers.This reporting arose signi cantly over time (from 75.0% in the early period to 80.0% in the late period (P = 0.05, Table 2).On average, 84.7% of the RCTs reported that they received IRB approval, and the frequency of IRB reporting did not rise over time.In addition, blinding was performed in 55 (76.4%) of the RCT reports (from 1998 to 2017).The reporting of blinding did not rise signi cantly as the study period progressed from early to late periods (from 75.0% to 72.0%; P = 0.94).Thirty-nine (54.2%) of the total reports described the allocation concealment (P = 0.82, Table 2).
3. Qualitative change in RCTs over time 1) Jadad scale: RCT quality did not increase from the rst RCT publication in 1998 to all other publications up to 2017.

3) Cochrane collaboration risk of bias tool:
According to the CCRBT analysis, only one (12.5%)low risk of bias RCT was published between 1998 and 2002.From 1998 to 2007 and 2013 to 2017, zero (0%) and four (16.0%)RCTs on CRPS, respectively, were published that had a moderate risk of bias (P = 0.24, Table 3).

4) Association between RCT quality and other factors
The RCT papers were divided according to whether the funding source, IRB approval, con ict of interest, and sample size were described or not.The papers that described the funding source had signi cantly higher quality, as determined using the Jadad scales (P = 0.03, Table 4).The papers that described sample size were of signi cantly higher quality, as determined by the Jadad and van Tulder scales (P < 0.01), respectively, and were signi cantly more likely to be low risk of bias reports, as determined by the CCRBT (P < 0.01, Table 4).

Discussion
CRPS is a neuropathic pain disorder that occurs after tissue injuries and is known to cause tissue deterioration over time.Patients may present various symptoms, including severe pain, but the disease has been di cult to identify.Management of complications and symptoms has been so diverse that many different attempts have been made, but there has never been a standard treatment.Although the mechanism of the disease is not clear, it has been mainly treated with a variety of drugs, but recently, various methods, such as rehabilitation and nerve block, have also been tried.
The present analysis of all 72 RCTs on CRPS revealed a quantitative and qualitative improvement over time.However, the degree of improvement was suboptimal, and the identi ed de ciencies should be addressed in future studies.
Several previous investigations have focused on the issue of RCT quality (12,21,22,24,25).They found that the number of RCTs published in journals from the 1980s to the 2000s increased over time.They also found an improvement in RCT quality over time (24).However, the authors applied the Jadad scale only and no statistical analyses of the data were done.To address these issues, previous studies showed a quantitative increase in RCTs over time (21,22).In the present study, we saw a recent increase in the number of RCTs published from the analysis of outcomes at 5-year intervals.
To evaluate adherence to the CONSORT statement, Uetani et al.( 4) identi ed 98 RCTs and reported that only 11 of these 98 RCTs adhered to each item of the CONSORT statement.However, since the CONSORT statement is not a quality assessment tool, no statistical analysis of each article's quality was possible.There are many tools for quantitatively assessing the quality of RCTs such as Moher (25), Chalmers (26), Jadad (12), van Tulder (14), Cochrane (15) and others.In this study, we used three tools, which were the Jadad scale, van Tulder scale, and CCRBT, which can comprehensively analyze the various elements of the CONSORT statement.
In an analysis of the methodological quality of RCTs published in Rheumatology International from 1981 to 2012, Lee et al. (21) con rmed that there was a quantitative increase in quality.However, the authors reported that the increase in quality was suboptimal.To ful ll the criteria for a high quality RCT, Lee et al. (21) proposed that authors should improve the reporting of allocation concealment, generation of randomization sequences, design of double blinded studies, and IRB approval.In the present study, double-blinded studies accounted for 76.4% of all RCTs published between 1998 and 2017 (Table 2).As in previous studies, inadequate explanations of blinding and the absence of any references to allocation concealment were factors that caused the evaluation of an RCT to be poor quality.Hewitt et al. (27) reported that 46% of the RCTs published in 2000 in the world's four major medical journals (The British Medical Journal, The Journal of the American Medical Association, The Lancet, and The New England Journal of Medicine) had involved inappropriate or uncertain concealment of allocation.Moreover, Schulz et al. (23) suggested that without concealment of allocation, randomization could be compromised and distorted by more than 40% in the process of performing even if the randomization procedure is appropriate.Based on the present study, appropriate blinding and allocation concealment are associated with RCT quality, therefore, if the quality of these items are improved, the RCT quality will also be improved.
As shown in previous studies, articles published after IRB review had a relatively high quality (22).IRB approval is a valuable step in terms of ensuring valid study design and performance, and the bene t of establishing a study plan for the IRB approval process is likely to be shown in the observed increase in the publishing rate of high quality articles.28) hypothesized that RCTs supported by funding would be of higher quality than those without funding, as they would be large-scale and well-designed.However, they found no association between funding source and article quality in an analysis of 100 RCTs published in a total of ve high impact, peer-reviewed general medical journals.We investigated articles related to CRPS published over the past 20 years in the pain medicine eld.Contrastingly, funding sources signi cantly impacted RCT quality, which was evaluated with the Jadad scale in the present study (P = 0.03, Table 4).In the case of research that received nancial support, it is judged that there are many high-quality articles because it is possible to do well-designed research design and large-scale research.

Clifford et al.(
Recently, many editors have requested clari cation on the number of sample size of the research subjects.In this study, about 37 articles (51.4%) based on the Jadad method provided the basis for the sample size of the study subjects, of which 34 studies (91.9%) showed signi cantly higher quality than those which did not provide sample size calculation (P < 0.01, Table 2).The value of the van Tulder scale was 7.59 ± 1.44, and 36 studies (97.3%) showed a high quality evaluation, showing a signi cant difference (P < 0.01, Table 2).
In recent years, there has been a debate about whether the con ict of interest should be viewed as a new bias evaluation area or not (29).However, it is necessary to examine factors that have not been agreed to yet, and that may have an effect on bias.
Future RCTs should address the complexity of CRPS, which has the characteristics of spontaneous pain, allodynia, hyperalgesia of the limbs, and abnormal ndings in the autonomic and motor nervous systems.CRPS potentially affects almost all systemic organs.Risk factors include smoking, genetic factors, psychological factors, and other factors that have not been yet been detailed.In 1994, the International Association for the Study of Pain (IASP) established diagnostic criteria for CRPSs.However, even after the development of the diagnostic criteria, disagreements persisted.There are two types of CRPS.Type 1 CRPS is not associated with obvious nerve damage and most patients belong to Type 1.There is no de nite cure, and several therapies have been tested.
Although several RCTs have been reported for CRPS, the number and quality of papers may be lessened due to the rarity of the CRPS disease itself and the di culty in designing the research.No de nitive pathophysiology or treatment has been established yet for the disease.Therefore, the quality of the RCTs related to CRPS should be assessed for completeness and the information of each article should be validated.
This study had some limitations.As noted in previous RCT qualitative studies, no consensus has yet been reached concerning the optimal method of quality assessment, and no highly accurate and valid tools for quality assessment have been established (30).To overcome these limitations, the present study applied three different tools: the Jadad scale, the VTS, and the CCRBT.
The extraction and quality evaluation of RCTs may be in uenced by the subjective judgement of the researcher.In the present study, thus, two independent researchers were responsible for data extraction and evaluation.Moreover, the data was evaluated and adjusted by a third researcher to optimize objectivity and reliability.To our knowledge, the present study represents the rst systematic evaluation of all RCTs on CRPS published to date.The results suggest strategies to improve the quality of research on CRPS.

Conclusion
The present data suggests that qualitative improvements for RCTs on CRPS are necessary.Even though double blind study designs and allocation concealments were performed in many of the selected RCTs, the quality of the RCTs was found to be suboptimal.However, sample size calculations, which recently editorial committees have increasingly requested authors to report, appeared to be a signi cant factor in improving the quality of RCTs.To achieve a signi cant improvement, pain practitioners and medical researchers require increased awareness of the criteria of the CONSORT statement, and future RCTs should involve funding sources, adequate randomization, blinding, IRB approval, and maintenance of allocation concealment.
We suggest that sample size calculations should be a possible new component of quality assessment tool in future studies.

Abbreviations
Evidence-based Medicine (EBM), Randomized controlled trial (RCT), Consolidated Standards of Reporting Trials (CONSORT), Van Tulder scale (VTS), Cochrane Collaboration Risk of Bias Tool (CCRBT), Complex regional pain syndrome (CRPS), institutional review board (IRB), N-Methyl-D-aspartate (NMDA), International Association for the Study of Pain (IASP) Declarations Ethics Approval and Consent to Participate : Not applicable Consent for Publication : Not applicable