In this study, we performed a survey to assess the guieline adherence of TCM pilot trials. The guideline adherence varied acrossing the checklist items, where some items required significant improvement. The guidance papers published in 2016 seemed to exert minimal effect on guideline adherence in TCM pilot trials. We also identified several issues specific to TCM pilot trials in this sruvey.
Interestingly, there were only 24% TCM pilot trials that had an objective of feasibility and were performed in preparation for future definitive trials (FDT). This indicated the inappropriate use of the term pilot in many small trials that aimed to test the hypotheses of effciacy or safety with an insufficient sample size albeit being underpowered to do so [8, 14, 15]. It also corresponded to the item 2b (specific objectives or research questions), where surprisingly only 3 (25%) FDT clearly stated their objectives related to feasibility. Furthermore, there were only two items (7a and 20) found with significant improved guideline adherence in FDT compared with NFT, implying that more endeavours were requried even in those pilots trials with specified feasibility objective(s). Therefore all these findings suggested further dissemination of the guidline to help clarify the definition of feasibility and pilot trials [2] and to enhance the guildeline adherence [3].
Likewise, our study indicated that the impact of CONSORT extension for pilot trials warranted more efforts in TCM pilot trials, because the improvement was only found in one item (12a) after the guidelines were published (Table 2). The minimal effect of the guidance papers may be because either that the guidelines did not reach the relevant research parties, or that the guidelines were largely ignored by the research parties [8]. In any case, our survey reveals the urgent need for both training and dissemination of research methodology and guideline adherence in TCM pilot trials.
Besides the common practice of the inappropriate hypothesis testing and insufficient power for conclusion in pilot trials [14, 16], our study also identified some issues specific to TCM pilot trials including blinding, standards for intervention and comparisons, and bias reduction (Table 3). This entails more guidance on methodology and reporting specific to TCM pilot trials, because the existing guidelines including CONSORT extensions to acupuncture [17], herbal interventions [18], and PAFS [3] could not fully cover these issues in TCM pilot trials. The progression criteria (guideline adherence level: 4%), sample size rationale (18%) and generalizability of methods and findings (18%) were also notable issues found in the TCM pilot trials (Table 2). This may be, at least in part, due to the insufficient details on explanation and elaboration from the guideline. For example, even though the CONSORT extension recommended that authors should justify the number of pariticipants in pilot trials [3], no sufficient details on how to exactly provide sample size rationale could be found in the guideline. Likewise, how to specify the progression criteria to determine whether the pilot trial can progress to future main trial, and whether the methods and findings can be generalizable to main trial and other pilot studies, required further detailed investigation and guidance in TCM pilot trials. The TCM field is substantially different from modern medicine, especially in their intervention, control and outcome assessment. Thus, our findings call for the need for further methodology and guidance in the research area of pilot and feasibility studies to address the unidentified issues and the other notable issues related to TCM pilot trials.
Our study was the first to explore the current practice of methodology and reporting in TCM pilot trials. We completed the data acquisition and analyses by two reviewers independently, thereby enhancing the accuracy of study findings [19]. There are also some limitations to our study. Due to the small numbers of the included FDT (n = 12) and studies published after year 2016 (n = 12), we only performed raw comparions without adjustments, which may yield biased findings in univariate analyses. We could not further extract potential solutions from the included TCM studies, indicating the important gap in methodological guidance in TCM pilot trials. Furthermore, only studies in Chinsese and English were screened and selected, which may therefore introduce selection bias due to lack of studies in other languages such as Japanese and Korean. Moreover, the impact of time lag between the publication of a new guideline and the adoption and implementation of it could not be fully assessed, which may therefore weaken the findings of our study.
To conclude, the current practice in TCM pilot trials requried substantial improvement in the literature. The guideline seemed to have only minimal effect on the methodology and reporting in TCM pilot trials, and some issues related to TCM pilot studies still warranted further methodology and guidance. Further endeavors are needed for training and dissemination of guideline adherence, and development of more detailed methodology in the field of TCM pilot trials.