This was a scoping bibliometric review of 367 behavioral-focused preliminary intervention studies published between 2018–2020. We examined the usage of guidelines, checklists, frameworks, and recommendations (GCFRs) related to preliminary studies and determined associations between GCFR usage and the reporting of feasibility indicators and feasibility-related characteristics. Citing two or more GCFRs was associated with reporting a greater number of feasibility indicators and framing the study findings from a feasibility perspective. These data demonstrate the use of GCFR, as inferred from citations within reference lists, have a clear positive impact on the overall comprehensiveness of the information presented. This information, in turn, should lead to greater transparency in reporting and more informed decisions regarding the viability of a behavioral intervention in a larger-scale trial.
To the authors’ knowledge, this is the first review to document and analyze the usage of GCFRs in preliminary health behavior preliminary intervention studies. Literature on Reporting, Design and Interpretation, and Defining Pilot and Feasibility Studies made up the majority of citations and the three most cited GCFRs were the Medical Research Council (MRC) Guidance for Developing and Evaluating Complex Interventions (13, 20), the CONSORT Extension for Pilot and Feasibility Studies (8), and the Template for Intervention Description and Replication (TIDieR) (21). According to the Web of Science (accessed April 14, 2022), the MRC guidance has been cited over 4,000 times, the CONSORT Extension over 500 times, and TIDieR over 3,000 times, although preliminary studies do not account for all of these citations. It is not surprising though that these are the top cited GCFRs in our sample of preliminary studies as each of them are widely supported in the behavioral sciences, funding agencies. These GCFRs can also be found on the EQUATOR network, which makes them easily accessible to a larger audience of researchers developing intervention studies.
Less common were GCFRs related to Adaptations and Scale Up, with the least commonly cited GCFR category being Progression Criteria. This might explain the very low presence of progression criteria in our included sample of preliminary studies, which has been found in other reviews as well (23, 24). The low usage of GCFRs related to Scale-Up is interesting, especially since the majority of preliminary studies are presumably designed for future scale-up. Several frameworks, including the National Institutes of Health Stage Model (25) and the Obesity-Related Behavioral Intervention Trials (ORBIT) model (26), outline the sequential and iterative processes for developing large-scale interventions, and specific attention is devoted to preliminary studies as a foundational piece of this scale-up process. Many of the GCFRs highlighted in this review can provide useful guidance on the scale-up process.
The reporting of feasibility indicators and feasibility-related characteristics in the sample of preliminary studies included in this review is similar to that of other reviews of health behavior intervention studies that are not preliminary studies (11, 24, 27). Most studies did not report feasibility indicators such as treatment fidelity, data collection feasibility, cost, participant compliance and attendance, and adverse events. Overall, most studies reported two or fewer feasibility indicators, which were typically recruitment and retention of participants. While trial-related feasibility (i.e., recruitment and retention) is important to assess, intervention-related feasibility indicators (treatment fidelity, the ability to collect data on participants, etc.) are equally important to measure and report to highlight the potential viability of an intervention and/or what aspects of the intervention need to be addressed and altered to ensure the intervention is successfully scaled-up in the future.
While most studies did mention feasibility (or use similar language) in the title and purpose statement, over half mentioned efficacy in the purpose statement and conducted statistical testing for efficacy. It is well established researchers should exercise caution when conducting efficacy testing within preliminary intervention studies and should not depend on these estimates to inform the design of a larger-scale trial, which would likely be conducted with hypothesis testing in mind (28–30). In fact, many of the GCFRs identified for this review caution against the use and interpretation of preliminary study effect sizes. This could explain why the use of GCFRs was associated with lower odds of many efficacy-related study characteristics in our sample. There are also inherent issues (both Type I and Type II error) with using preliminary study effect sizes for sample size estimation for a larger-scale trial, although we did not identify whether this was being done in our sample of preliminary studies. It is worth noting that just over half of the included preliminary studies did advise caution regarding interpretations of efficacy and most studies framed feasibility indicators as the primary outcomes of interest. This is a promising finding and shows that many authors of recently published preliminary studies are 1) acknowledging the pitfalls of using preliminary studies to demonstrate the efficacy of an intervention and 2) prioritizing feasibility indicators as the main outcome of interest.
While this study is not the first to explore reporting quality in preliminary studies (11, 24, 27), it is the first to investigate associations between the usage of GCFRs and preliminary study reporting. Some studies have broadly explored the impact of guidelines (mostly CONSORT and TIDIER) on study reporting quality, not necessarily in preliminary studies exclusively, and the results of these studies are heterogeneous (31–33). There are also several other examples of these types of reviews for a variety of disciplines and study designs (34–38), but most use somewhat ambiguous proxies for the usage of guidelines. In other words, most reviews have assessed reporting quality before and after the publication of a guideline, with the assumption that the presence of the published guideline might influence reporting quality. Our study utilized text mining to identify preliminary studies that cited GCFRs, using the presence of a GCFR citation as the indicator for usage and found that usage significantly and positively associated with the reporting of most feasibility indicators and feasibility-related characteristics. These findings provide compelling support for the use of GCFRs as a tool to improve the reporting quality of preliminary intervention studies.
Not only does the usage of GCFRs (via citation) associate with better reporting, but our results in the Tertiary Citer group show citing a different preliminary study that cited a GCFR associated with increased odds of reporting acceptability and treatment fidelity as well as decreased odds of conducting statistical testing for efficacy. These results demonstrate a possible diffusive nature of the information published in GCFRs, whereby authors use other published preliminary studies (which cited a GCFR) as a model for their work. Thus, the true beneficial impact of GCFRs related to preliminary studies may go beyond just improving the reporting quality of studies which cite them. But their reporting was not a much as Citers – correct?
Strengths and Limitations
This review included a large sample (N = 367) of preliminary intervention studies published between 2018–2020, capturing some of the most recently published health behavior preliminary studies in the literature. Preliminary studies were not excluded based on location, design, or health behavior topic, which means results are generalizable to a large audience of health behavior researchers who conduct preliminary intervention studies. However, several limitations need to be considered. Included studies came from a sample 25 journals publishing the largest number of preliminary studies. Sampling from a wider variety of journals may have produced different results. Because the number of studies not citing any GCFR (Non-Citers) was so large, we opted to randomly sample 200 of them. Although random sampling is supposed to be an unbiased approach to sampling and no differences between the subsample and larger sample were found in our study, there is a possibility of the presence of sample selection bias. Limitations regarding the coding of studies are also important to mention. First, the presence/absence of GCFR usage and the reporting of feasibility indicators and feasibility-related characteristics was identified via a combination of text-mining and manual approaches. It is possible that some items may have either not been identified or improperly coded. While we used an extensive list of GCFRs, it is possible that some GCFRs were missed and therefore not searched within our sample, which could influence results by miscoding studies that did cite a GCFR which we did not include in our search. Finally, we recognize that what is reported in a study is not completely identical to the actual conduct of the study. The failure to cite a GCFR or report a feasibility indicator does not mean authors of a study had not considered preliminary study guidance or measured feasibility indicators during their preliminary intervention study.