In the current project we assessed the research impact on clinical practice of publicly sponsored trials and commercially sponsored trials conducted in Germany compared to those conducted internationally. By using a prospective strategy that followed the lifecycle of trials from their registration up to their inclusion in systematic reviews and clinical guidelines, we have collected and analyzed data not only for those trials that were ‘successful’, but also for trials that were not published and did not gain impact on clinical practice.
Interpretation of finding
We compared IITs and IST because they often focus on different clinical questions and pursue different aims and objectives. IITs play a crucial role in academic clinical research whereas ISTs usually focus on commercial interests, mainly of pharmaceutical companies, whose primary aim is to develop and approve drugs or other medical treatments [29]. In IITs, an academic investigator is responsible for the conduct of the clinical trial, which includes planning, registration and publishing the results of the study [30]. IITs are often conducted to expand product knowledge, including safety, and to identify new ways of using existing treatments, which might lead to the improvement of patient health [31]. IITs complement ISTs regarding the medical field, such as physio- and psychotherapy, behavioral changes and complementary medicine.
Compared to literature, in our project a high proportion of trials (83%) were published. Systematic reviews and retrospective research projects investigating the publication proportion of RCTs resulted in considerably lower proportions of 60% to 71%.[1, 32, 33] Only 37% of RCTs presented as conference abstract were published in full as journal article [2]. The relatively high proportion of published trials and trial results in our cohort can be attributed to the fact that we also considered a trial as published, when its results were reported in a study registry.
Even though there are several advantages for posting results in registries, e.g. results can be presented fast and concisely, they are directly attached to the registry record, providing information about the trial methods as well as references and links to further trial information, the publication proportion is still relatively small. In a cross sectional study across academic medical centers in the United States, the publication proportion of completed trials that were registered in ClinicalTrials.gov was analyzed. Across the medical centers, 10.8% to 40.3% of the trials were published within 24 months of study completion, and for 1.6% to 40.7%, results were reported on ClinicalTrials.gov [34].
In our cohort, on average, for 45% (range of sub-cohorts: 36%-68%) of the trials we found results in study registries. This finding is in line with the results of a recent study, investigating the compliance with the Food and Drug Administration Amendments Act of 2007 (FDAAA) concerning reporting of trial results. The researchers found, that only about 40% of all applicable trials due to report results under the FDAAA reported their results in ClinicalTrials.gov within the one year deadline after study completion [35].
The possibility to add study results to a study in registrie, is certainly an important step to improve transparency in clinical research, but limited, incomplete or expired trial information in registries often make it difficult to get a complete picture of the trial and to appraise and interpret the results. Several initiatives such as AllTrials and TranspariMed work on the improvement of a trial’s reporting by requiring clinical trials to be registered and to report their full methods and summary results [19, 20].
In our cohort, of the trials with published results, 19% (107 of 556) were solely available in study registries. This has serious implications for the search process to identify relevant studies, i.e. which sources need to be searched, especially for systematic reviews and clinical guidelines. A search strategy should not only focus on journal articles, but should be accomplished by an additional search in study registries, which has already become mandatory for conducting Cochrane intervention reviews [36, 37]. To improve the findability of trial results, the registries themselves should improve their searchability. They should be constructed in a standardized format so that they are easily and reliably searchable, e.g. similar to biomedical databases by title, author, keywords and abstracts. Looking at our sub-cohorts, we found a significant difference between IITs and ISTs. For publicly sponsored trials, only 2%-4% were solely published in registries, whereas this was the case for 20% to 32% of the commercially sponsored trials. The reasons for this difference are unclear and future analyses would be worth to compare the characteristics and results of those trials published solely in registries with those published as journal articles, e.g. regarding publication bias. One explanation could be that for publicly funded trials publication of results in the form of a journal article is often demanded by the funding organization and is part of the funding conditions. Advantages of publishing results as a journal article ideally are a quality-assured peer-review, trial methods and results are considered and discussed in the context of the existing evidence and can be commented by other researches e. g. via response letters. For a great proportion of trials in our cohort, results were published as journal article (81%), and for more than half of the trials results were included in a registry.
Disclosure of detailed trial methods of a trial is essential with respect to the critical appraisal and interpretation of the results, and is the basis to enable other researchers to reproduce the trial and verify its results, which is a basic requirement for later implementation in medical practice. While in an original journal research article both methods and results of a trial are described, it is becoming more common to publish articles only describing the detailed methods of a trial and not the results. In our cohort, this was the case for 14% of the trials, for 3% only a method article could be identified. Moreover, it is important to point out that most of the method articles derive from publicly funded trials (87%), of which most of the German IITs were from Public Germany gov (67%). In scientific research it is not unusual to publish results of one study in more than one article. One reason for this could be that in academia the reward system is often built on quantity of research output [38]. Scientific success, such as reputation, career advancement, successfulness of applications for research funding, is directly associated with the publication output of a researcher. In our project, multiple publication was the case for 188 (40%) trials. They were more common for IITs (Germany: 33%, International: 32%) than for ISTs (Germany: 17%, International 14%). The trial with the highest number of identified publications (n=21) was a phase 4 study, conducted in the field of cardiovascular disease, funded by the German Research Foundation. For this trial, one method article, two result articles and 18 sub-studies and secondary analysis were published. This trial and also the other high-frequently published trials (25 with more than 5 published articles) were conducted in academia. For this publication frequency, measured as the number of published articles per trial, we found a remarkable phenomenon: about one third of all published articles corresponded to only 8% of the trials. Even though this aforementioned reward system and its consequences have been in the focus of criticism for several years, structures have still not changed [39].
In contrast, in industry the central (financial) interest lies in the results, i.e. efficacy and safety of the tested treatment, whereas the study protocol and methods used are often confidential to protect commercial interests. This is also shown by the public availability of the study protocols: for 40 trials of our cohort, the original study protocol could be identified, 30 belonged to the IIT sub-cohorts and 10 to the IST sub-cohorts.
A Health Technology Report was conducted to evaluate the impact of Cochrane Reviews published by 20 Cochrane Review Groups, on health care, patient outcomes and value for money [40]. A random sample of 20 Cochrane Reviews and 40 selected reviews, more likely to have had an impact, provided by the Cochrane Review Groups were selected for further evaluation. “Overall, 40 of the 60 reviews had been cited in some form of clinical guidance and 15 had influenced further primary research.”
We found that more than half of the trials are represented in systematic reviews and more than a quarter in guidelines. To further improve this knowledge transfer from research into practice, several issues have to be considered. The first issue lies in why 17% of the trials have not been published. The second issue is how to improve the transfer of 28% of the trials that have been published but had no impact.
Typical reasons for not publishing trial results as presented in a systematic review are lack of time and/or resources, non-completion of study, publication was not an aim, or only had low priority [41]. Possible explanations for published trial results not being included in systematic reviews are that no review related to the research question has been conducted or updated after the date of publication. Reasons reported for non-inclusion of published articles in the systematic reviews of our study cohort were that the eligibility criteria were not fulfilled, e.g. wrong patient group, intervention, comparator, outcome measure, or study type.
For inclusion of trial results in guidelines, the same reasons as for systematic reviews could apply. However, in guidelines, in addition to publications, systematic reviews are also a relevant pathway for inclusion of trial results. A detailed investigation of the systematic reviews that have not been included in guidelines (56%) would be useful, e.g. to find the reasons for lack of guidelines and to be able to further improve the transfer of important trial finding into medical practice.
Strengths and limitations of the study
A strength of our study was that all trials were registered in study registries so that for all of them basic study information was available. Study characteristics were double-extracted independently in a pre-piloted extraction form following a written manual. All data extractors were trained. We captured all relevant information available in any study registry. Discrepancies in different sources were discussed and resolved. The identification of systematic reviews citing the original study report was conducted semi-automatically by using a self-developed program. The search for clinical guidelines was done manually following predefined standardized rules.
Another strength was that we controlled for possibly biasing factors by design, i.e. by balancing important study characteristics to Public Germany as the reference sub-cohort.
A limitation arising from this was the limited number of studies in the sub-cohorts Public Germany and Commercial Germany. The number of trials meeting our inclusion criteria for the Public Germany gov (reference sub-cohort) was fixed to 60. For the sub-cohort Commercial Germany, a balancing for non-drug trials was not fully possible: only 171 could be identified in DRKS and ClinicalTrials.gov registries instead of the pre-planned 200 studies per comparison sub-cohort. However, it is not expected that this difference of 29 trials have a relevant influence on our results.
Our cohort was composed of trials that were included in study registries and, partially, also in databases maintained by funding organizations. Against the background that still not all trials are registered, our trial cohort might be a “positive” selection compared to those conducted worldwide. Therefore, there is a potential risk that our cohort is biased, resulting in a limited external validity of our project results.
Even though all studies were included in at least one study registry, for some studies information in registries was scarce and detailed study protocols were only rarely available. Therefore, for some trials it was difficult to find out whether a published article corresponded to the trial. We also had to rely on the information reported in registries. Data of prospectively registered studies can include preliminary study information, for example information about study start and completion date. This may have influenced our findings.
We tried to assess actual data and included trials that started in 2005 or later and were completed by the end of 2016, for which we searched for corresponding publications in 2018/2019. For trials completed late during this time period, there might not have been enough time for publication and inclusion in systematic reviews and guidelines. Our results, however, indicate that this only concerns a few trials because since 1) compared to literature, the publication rate of our cohort was relatively high, and 2) the stacked probability plot (Figure 3) also indicates that only few first publications are to be expected. Nevertheless, in such projects there will always be a compromise between presenting actual data with respect to the timeframe of included studies and leaving enough time for studies to be published and have an impact.
An important result of our study was that for 15% of the trials, results were solely available in study registries and were not published as journal article. In such cases, we could only search for guidelines citing the trial by using the registry identification number, but this was not possible for systematic reviews. To identify citing systematic reviews, we used the “cited by”- or “times cited”-functions of PubMed and WoS. These functions only consider journal articles, so that we were limited to the published journal articles.
The full text of some clinical practice guidelines from the United Kingdom identified via NICE or TRIP were only accessible to people located within the country, so that we were not able to verify the citation for those. Therefore, we did not consider them for our project.