This was a historical scoping review of the reporting of feasibility indicators in a large sample of behavioral pilot/feasibility studies published between 1982–2020. We describe trends in the amount and type of feasibility indicators reported in studies across three time periods evaluating 200 studies from each period: 1982–2006, 2011–2013, and 2018–2020. Improvements over time were found for the reporting of most feasibility indicators; however, the rates of reporting remain modest, even in the latest group of studies published from 2018–2020. The majority of behavioral pilot studies reported three or fewer feasibility outcomes, the most common being recruitment and retention, while almost all studies conducted/reported statistical analyses related to preliminary efficacy.
The primary finding from this study was the suboptimal rate of reporting key feasibility indicators within behavioral pilot/feasibility studies. While trial-related feasibility was reported in the majority of studies (recruitment and/or retention), key intervention-related feasibility indicators, including participant acceptability, adherence, attendance, and intervention fidelity were not widely reported. These results are supported by several reviews of pilot/feasibility studies conducted in other domains,33,34 which all found a lack of trial- and intervention-related feasibility indicator reporting as well. While recruitment and retention are important trial-related feasibility indicators to capture, intervention-related feasibility indicators are important to assess during the preliminary phases of implementation. For example, participants’ perceptions of programs (acceptability) are associated with rates of attrition,35 intervention attendance is positively associate with health outcomes,36,37 and implementation fidelity during a pilot/feasibility study is associated with main outcomes in scaled-up trials38–41 and is shown to moderate the association between participant acceptability and behavioral outcomes.42
The lack of reporting feasibility indicators coupled with the high rate of statistical testing for preliminary efficacy is concerning as well, although this does seem to be common across domains. For example, in a review of nursing intervention feasibility literature, Mailhot et al.33 found that almost half of the included feasibility studies focused exclusively on testing effectiveness. While preliminary efficacy can be reported in pilot/feasibility studies, results should be interpreted with caution, and outcomes related to feasibility should take priority.
Results from our study suggest that this is largely not the case in the behavioral sciences and reasons why remain unclear. It could be that intervention funders are invested in the outcome data. In other words, those agencies which fund preliminary studies might want some evidence that the intervention will have beneficial impact (regardless of its precision) before they continue to invest considerable time and money in a large, definitive trial. Several published guidelines, checklists, frameworks, and recommendations for pilot/feasibility studies exist, 2,5,7,10,11,43−45 many of which argue against the use of and focus on statistical testing for preliminary efficacy. However, pilot/feasibility studies have only just recently garnered attention from larger agencies. For example, the CONSORT extension to randomized pilot and feasibility trials10 was published in 2016 and the majority of other literature that is used to guide pilot/feasibility studies has been published within the last decade as well. For pilot/feasibility studies included in this review, the most commonly cited guidelines/frameworks included the Medical Research Council guidance1, the CONSORT extension for pilot and feasibility studies10, and the RE-AIM framework32. Other guidelines used less often included Bowen et al.,12 Thabane et al.5, and Arain et al.28 Researchers conducting preliminary studies today are encouraged to use the available literature in an effort to design high quality preliminary interventions that can provide rich data to support the successful scaling up to a larger trial.
While our review does highlight some concerns for behavioral pilot/feasibility studies, there were also some encouraging findings. We found studies conducted between 2018–2020 had higher odds of reporting most feasibility indicators when compared to studies published between 1982–2006. While reporting was still only modest in the later studies, results do show that improvements are occurring among behavioral pilot/feasibility studies. This may coincide with recent initiatives that have been undertaken in the field, including the publishing of several frameworks, guidelines, and recommendations related to pilot/feasibility studies.2,5,7, 10–12,18,44,45 Our results demonstrate that the reporting of feasibility indicators positively associated with citing a guideline/framework for the reporting of preliminary studies. Researchers conducting pilot/feasibility studies should utilize these guidelines/frameworks to inform the design, conduct, and reporting of their preliminary work, as our results support the idea that these guidelines/frameworks can improve the completeness of reporting in pilot/feasibility studies. We also found that the reporting of feasibility indicators positively associated with mentioning feasibility-related outcomes in the purpose statement of the published pilot/feasibility study. This may demonstrate the importance of stating clear objectives. Alternatively, it may also suggest that authors of these papers were generally more sensitized to the need to be explicit about aspects of feasibility.
The reporting of feasibility indicators also significantly and positively associated with a study being supported by funding of any kind. It is well established that pilot/feasibility studies play an essential role in the development of larger-scale trials and virtually all funding agencies require evidence gathered from these preliminary studies to support the justification for scaling up to a larger trial. This highlights the importance of funding structures that are designed to support the conduct of pilot/feasibility studies specifically. Recent initiatives like the NIH Planning Grant program (R34),46 the CIHR Health Research Training Platform (HRTP) Pilot Funding Opportunity,47 and the NIHR Research for Patient Benefit (RfPB) program48 represent important steps forward in the field of pilot/feasibility research.
Strengths and Limitations
A strength of this review is the inclusion of a large sample (N = 600) of obesity-related pilot/feasibility studies published across four decades. Even though this was a scoping review and not every study published between 1982 to 2020 was included, we did not limit the inclusion of studies based on location, design, or health behavior topic, as long as the intervention contained at least one component related to obesogenic behaviors. As such, results can be generalized to a larger audience of health behavior researchers. There were also limitations to this review. First, we only considered health behavior interventions related to obesity for inclusion. While results may generalize to pilot/feasibility studies in the realm of health behavior, they cannot apply to non-behavioral preliminary studies including mechanistic, pharmacological, or rehabilitation interventions. Another limitation is that studies in the Early Group span a much greater length of time (1982–2006) compared to studies in the Middle (2011–2013) and Late (2018–2020) groups and each year is not equally represented. This was a function of the limited number of pilot/feasibility studies published in earlier years compared to later years. It must also be noted that some feasibility indicators may not have been relevant to collect for certain intervention designs. For example, attendance would most likely not have been an applicable feasibility indicator for an mHealth intervention, but participant compliance may have been a feasibility indicator of interest. In other words, depending on intervention design and the specific components of each pilot/feasibility study, it would be impossible or irrelevant for some studies to collect 100% of the feasibility indicators for which we coded. Furthermore, some of the feasibility indicators are difficult to code. We used both text-mining and manual approaches to maximize accuracy in capturing this information, but some items may have been erroneous. Finally, reporting of a study is not identical to the conduct of a study. Not reporting of some aspect does not mean that the study authors had not taken it into consideration.