DOI: https://doi.org/10.21203/rs.3.rs-44819/v1
Background – While immune-checkpoint inhibitors (ICIs) have transformed the field of oncology for advanced stage cancers, they can lead to serious immune toxicities. Several systematic reviews have evaluated the risk of immune-related adverse events (irAEs); however, most have focused on published articles without evaluating trial registries. The objective of this methodological review was to compare the reporting of safety information and in particular, serious irAEs (irSAEs), in both publications and ClinicalTrials.gov for all current FDA-approved ICIs.
Methods – MEDLINE was searched via PubMed to retrieve all published phase III randomized controlled trials (RCTs) evaluating ICIs. For each eligible trial, we searched for corresponding registration on ClinicalTrials.gov and extracted relevant safety data from both the publication and results posted on registry. We then compared reporting and evaluated concordance in reported safety data between both sources.
Results – Of 42 eligible published trials, 34 had results posted on ClinicalTrials.gov. Considerable variability was noted in the reporting of safety in both sources. SAEs were reported for all trial results in ClinicalTrials.gov compared to 23.5% of publications. An overall incidence for irAEs and irSAEs was reported in 58.8% and 8.8% of published trials respectively, compared to 11.8% and 5.9% of registry results. Evaluating the concordance of specific irSAEs was not possible between the two sources in 32/34 trials either due to different reporting formats (61.8%) or data not being reported in one or both sources (32.4%). From the 2 studies with compatible irSAE format, only 1 had concordant data between both sources.
Conclusions – The reporting of irAEs / irSAEs varies considerably in publications and registries, which outlines the importance of standardizing the terminologies and methodologies for reporting safety information relevant to ICIs.
Immunotherapies have transformed the field of cancer therapy by improving the overall prognosis of patients, especially for recurrent and metastatic cancers [1–3]. Immune checkpoint inhibitors (ICIs) are a type of immunotherapy which result in increased activation of the immune system, allowing it to recognize and destroy tumor cells [4,5]. However, ICI may also lead to potentially serious drug-induced immune toxicities collectively known as immune-related adverse events (irAEs), which, depending on their severity, may result in substantial declines in organ function and fatal outcomes [6,7]. The rapid increase in the number of medication alerts regarding irAEs received by regulatory authorities suggests that immune toxicities may constitute a competing event with cancer evolution, making the assessment of irAE and serious irAEs (irSAE) a major concern. Despite efforts to develop standardized definitions and guidelines for their recognition and management [8], the reported incidence of irAEs varies greatly between studies ranging from 15 to 90% [9,10] , which may be partly due to inconsistent and incomplete reporting or characterization of AEs within clinical trials [11].
While previous systematic reviews have evaluated the quality of irAE reporting in publications of ICI clinical trials [12] or assessed their incidence [11], most have not considered information from clinical trial registries which include key information from the trial protocol (registered prior to patient recruitment), as well as results posted following trial completion (i.e., participant flow, primary and secondary endpoints and all serious and non-serious AEs) As a result, registries are recognized as an important source of information when conducting systematic reviews, not only to identify unpublished trials and evaluate the risk of selective outcome reporting, but also to extract results, and in particular safety results [13,14]. This is while previous studies have showed that certain safety information such as serious adverse events (SAEs) were more completely reported at ClinicalTrials.gov than in corresponding publications [15,16].
Therefore, the primary objective of this study was to evaluate whether there are discrepancies in the reporting of safety and irSAEs in particular between clinical trial publications and corresponding results posted on ClinicalTrials.gov for current FDA-approved immune checkpoint inhibitors (ICIs) in oncology.
We performed a methodological review of the reporting of safety results focusing on immune-related serious adverse events (irSAEs) in publications and registries for all current FDA-approved ICIs (Appendix 1): CTLA-4 (ipilimumab), PD-1 (nivolumab, pembrolizumab) and PD-L1 (atezolizumab, avelumab, durvalumab and cemiplimab).
2.1 Terminology and definitions
A complete and detailed list of the following terms and definitions which have been used in this study are provided in Appendix 2: Structural hierarchy of adverse events, severity of adverse events, seriousness of adverse events, immune-related adverse event (irAE), and immune-related serious adverse events (irSAE).
2.2 Search for publications
A search in MEDLINE via PubMed was conducted to identify all randomized controlled trials (RCTs) assessing currently FDA-approved ICIs (Appendix 1). The search algorithm included key-words and free-text words for immune checkpoint inhibitor or blocker (anti-CTLA-4, anti-PD-1, anti-PD-L1) and drug names for currently FDA-approved ICIs and applied the Cochrane’s filter (sensitivity- and specificity-maximizing version) to identify RCTs (Appendix 3).
2.3 Eligibility criteria
Phase III RCTs for all FDA-approved ICIs used in cancer treatment which were published in English prior to March 2019 were included in this study. Phase I, II or IV trials, duplicates, abstracts of conference proceedings, case reports / series, editorials, commentaries, expert opinions, letters, narrative reviews, secondary reports, retrospective analyses, systematic reviews and meta-analyses or non-English publications were excluded.
2.4 Selection process
All references were evaluated for eligibility by one of the authors (ZK) with any doubtful publications considered upon evaluation and approval by a second author (AD). The screening procedure was conducted based on a two-step process: (1) title/abstract screening using Rayyan [17] and (2) full-text screening.
2.5 Search for corresponding registration on ClinicalTrials.gov
For each selected published trial, ClinicalTrials.gov was searched for the corresponding RCT using the NCT number when provided in the publication. When the registration number was not reported (which was not the case for any of the eligible trials), we planned to search the trial acronym or key elements of the trial to identify the registration. According to the Food and Drug Administration Amendment Act of 2007 (FDAAA 801), applicable clinical trials (trials with at least one site in the US) must submit trial results within 12 months after the primary completion date. We therefore evaluated whether results were posted within 1 year for those concerned by the law.
2.6 Data extraction
A structured data extraction form in Excel was used to collect the following information from publication and ClinicalTrials.gov for each trial, which was carried out in duplicate (ZK and SM) with any disagreements resolved through discussion and consensus:
2.6.1 From the published report:
2.6.2 From the registry results:
2.6.3 From both sources:
Evaluation of overall / general safety information
Evaluation of specific safety information: irAEs and irSAEs
Of note, AEs were considered as immune-related only when clearly indicated as such by the authors. In other words, similar trAEs as irAEs (e.g., pneumonitis or colitis) which did not have an underlying immune etiology were not considered in the assessment. Definitions for key terms are reported in Appendix 2.
2.7 Concordance of key safety data between publications and registry results
For each trial, general safety parameters (listed above) as well as the incidence of irAEs and irSAEs were compared between the publication and results posted on ClinicalTrials.gov. When this was not possible, the reason (unreported value, inconsistent reporting format, etc.) was noted.
We first studied the overall incidence for each safety parameter between the two sources for each arm using the following approach (which is graphically illustrated in Appendix 4) :
(1) Concordant: when the reported values matched between the two sources for all treatment arms
(2) Partially concordant: when the value matched for one / some arm(s), but not all arms of the trial
(3) Discordant: if none of the reported values in the treatment arms matched between the two sources
(4) Not assessable or comparable: if the value was not reported in one or both sources (along with indicating the missing sources), or if they were not presented in the same format (barring a direct concordance assessment)
The reported frequencies from the two sources were marked as a match, if the rounded percentages were within ±1% of one another.
After comparing general safety information, we then compared the incidence of specific types of irSAEs (e.g., pneumonitis, colitis, rash, etc) between the two sources, in each trial arm. The same approach was used to assess concordance between the two sources regarding irSAEs (Appendix 4).
When there were several publications for a given trial, only the article with a publication date closer to when the trial results were posted on ClinicalTrials.gov was considered. This was to ensure that differences would not be attributable to updates in posting new trial results in the registry (basically, we wanted to make sure that detected differences were not a result of comparing newer trial results posted in ClinicalTrials.gov to old published information). Also, if the investigators of a trial had published efficacy and safety outcomes in separate articles, the publication reporting safety results was selected for the purposes of our study.
2.8 Statistical analysis
Data analysis was descriptive. Frequencies and proportions are reported for categorical data, while median and interquartile ranges are presented for continuous data. Statistical analysis was performed using R software (v3.3.1).
3.1 Selection process of published trials and general characteristics
From the 790 references retrieved by the search, we identified 51 primary publications of phase III trials. An additional 9 references were excluded since some trials had multiple publications (Figure 1). Of the 42 included trials published between August 2010 and February 2019 (Appendix 5), the most common indication was metastatic non-small cell lung cancer (n=16, 38.1 %). The median sample size was 695 (IQR 497 – 925). All trials had a parallel design, 36 (85.7 %) had 2 study arms and 23 (54.8%) were conducted as open-label studies. The most common primary outcome was overall survival (n=20, 47.6%) (Table 1).
3.2 Identification of corresponding trials in ClinicalTrials.gov and registration status
Of the 42 published RCTs, all were registered and the NCT number was systematically reported in the article, however, only 34 (81.0%) had results posted in the registry when we conducted our search on ClinicalTrials.gov (May 7, 2019). Of the 42 trials, 37 (88.1%) had at least one US site. Of these 37 trials, 18 (48.6%) had posted results within 1 year, 13 (35.1%) posted results after the deadline of one year and 6 (16.2%) currently do not have registry results posted (have not reached the deadline at this time).
3.3. Evaluation of Safety Information
3.3.1. Overall or general safety information
The population analyzed was indicated in all publications but was not clearly reported in 12 (35.2%) posts at ClinicalTrials.gov. All trials evaluated safety in patients who had taken at least one dose of the medication (as-treated population). The number of participants analyzed in each arm was reported in both sources and concordant between the two sources in 32 (94.1%) trials.
All trials explicitly stated the use of MedDRA for coding AEs in their registry results compared to only 10 (29.4%) published trials. On the contrary, all publications noted using the NCI-CTCAE grading scale to report the severity of AEs, while only 4 (11.8%) trials reported similar grading of AEs in ClinicalTrials.gov.
In 36 (85.7%) publications, the authors did not report all AEs but only those reaching a threshold which varied across studies, ranging from events experienced by 2-3 patients to those encountered in at least 15% of participants. In ClinicalTrials.gov, a frequency threshold of 5% was used for reporting AEs in all trials while no threshold was used for reporting SAEs. The overall incidence of SAEs was reported in all 34 registry results whereas this information was reported in only 8 (23.5%) publications. More publications reported the overall incidence of AEs and trAEs (n=17, 50.0% and n=29, 85.3% respectively) compared to registry results (n=3, 8.8% and n=7, 20.6%). The number of deaths due to AEs was reported in 32 (94.1%) registry results compared to only 9 (26.5%) published trials. This safety parameter was concordant between the two sources in only 1 trial (2.9%), while 7 (20.6%) trials had discordant values (Figure 2).
3.3.2 Immune-related adverse events (irAE)
There was considerable variability in the terminology used for referring to irAE. Publications predominantly list them under immune-related AEs (n=16, 38.11%), whereas most registry results refer to them as immune toxicities (n=24, 70.6 %). A clear definition for irAEs was provided in 35 (83.3%) of the 42 published trials compared to 4 (11.8%) trial results from the 34 RCTS with results posted on ClinicalTrials.gov, with even fewer trials defining irSAEs in publications (n=19, 45.2%) and registry results (n=1, 2.9%) respectively.
All published trials which reported irAEs noted that drug-causality was adjudicated by the investigators and that they were labeled as immune-related regardless of whether the investigators attributed them to the treatment or not. Only 1 (2.9%) trial provided a distinction between immune-mediated AEs (imAEs) – AEs with an underlying immune mechanism not attributed to the ICI, and immune-related AEs (irAEs) – AEs with an immunogenic cause that were attributed to the ICI.
The overall incidence for irAEs and irSAEs were reported in 20 (58.8%) and 3 (8.8%) of publications respectively, compared to 4 (11.8%) and 2 (5.9%) of registry results (Figure 3).
For published trials, while the reporting format varied greatly depending on the level of structural hierarchy chosen by the authors – SOC, PT or both – all had indicated the use of the NCI-CTCAE grading scale for reporting the severity of irAEs. In contrast, all trial results posted on ClinicalTrials.gov reported the frequency of irAEs using PTs with only 2 (5.9%) of the 34 trials reporting using a grading scale for the severity of irAEs. Consequently, only 2 trials were identified as having a consistent reporting format to the registry for irSAEs, of which only 1 (2.9%) trial had concordant results between the two sources (Figure 4).
Table 2 summarizes the differences in formatting components relevant to the reporting of safety data (including irSAEs) between publications and ClinicalTrials.gov.
To our knowledge, this is the first study to provide a comparative assessment of the reporting of safety with a focus on irSAEs from ICIs in publications and registry results posted on ClinicalTrials.gov. There was a considerable variability in safety reporting across studies and between both sources. Despite the need to promptly detect irSAEs, published trials of ICIs and registry results from ClinicalTrials.gov do not consistently report irAEs or irSAEs with the rate of irAEs and irSAEs being reported in 58.8% and 8.8% of published trials respectively, as compared to 11.8% and 5.9% of trial results in the registry.
Similar to previous research, our study shows that certain safety results such as SAEs are more completely reported in ClinicalTrials.gov [15,16,22]. Since the reporting of all SAEs is mandatory in ClinicalTrials.gov, all trial results in the registry had reported SAEs while this key safety parameter was missing from 76.5% of published trials. It is worth noting that while grade 3, 4 and 5 AEs are all serious by definition, not all SAEs include events that are grades ≥ 3. This is because occasionally AEs grades < 3 (e.g., a grade 1 myocarditis or grade 2 rash) might require medical intervention for symptomatic management or prevention of further progression. As a result, the simple addition of the incidences of grades 3, 4 and 5 AEs does not accurately reflect the rate of SAEs in RCTs, which underscores the importance of reporting both the severity and seriousness of adverse events for investigational drugs.
Reporting SAEs (including death and hospitalization), in particular irSAEs - which are due to the drug’s mechanism of action – is crucial for ICIs given their labeled indication is for metastatic and recurrent cancers who are prescribed the drug with increased survival in mind [23–29]. Furthermore, there has been an increasing number of safety alerts due to such events in recent years [6] which frequently include severe irAEs that can be fatal. While ICIs may improve survival outcomes in patients with advanced malignancies, a significant proportion of patients will not respond and still have a poor prognosis [30]. More importantly, considering end-of-life comfort and quality of life measures as well as avoiding substantial treatment-related costs play crucial roles in determining treatment goals in terminally ill cancer patients [31]. Therefore, a more comprehensive evaluation of the overall incidence and type of SAEs in particular irSAEs associated with these medications will allow terminal / end-stage cancer patients and their physicians to make more informed decisions by determining whether the benefits of increased survival outcomes will outweigh the risk of death and impaired quality of life due to toxicity from these drugs [32].
With regards to specific types of irSAE (e.g., pneumonitis, colitis), we were unable to compare their incidence between sources in 94.1% of trials, mainly because of differences in the reporting formats used for presenting safety data (61.8%). The most variable factors between the two sources were the structural hierarchy level used for reporting each type of irAE (e.g., the incidence of all irAEs affecting the skin compared to autoimmune dermatitis) and the choice to report grading for the severity of AEs (e.g., the rate of serious autoimmune colitis compared to grade 3 and 4 colitis). The variability in the terminology used for referring to this particular class of AEs further complicates matters when cross-checking their incidence. Our results showed that there were various ways of referring to this specific category of AEs in publications and registry results, which will need to be standardized. If indeed these terms refer to different AEs, the differences should be clearly explained by the authors. This is especially an important next step for the incorporation of ICIs as part of standard cancer treatment modalities [33,34] since without the use of standardized terminologies and methods to consistently detect, collect, analyze and report irAEs [35,36], efforts to provide reliable estimations of irSAEs for each ICI medication and cancer type remain hindered.
As of 2011, both US regulations and the European Commission provided guidance placing a strong emphasis on early reporting of serious events with a reasonable possibility of being associated with the drug, so that safety analysis is not confounded by unnecessary reports of AEs which are not caused by the drug and that product safety can be assessed in a more rapid and meaningful manner [37]. Of the 42 published trials, 85.7% reported irAEs separately from other trAEs, however, the authors of only one trial (2.4%) provided a detailed explanation for how irAE causality was established with the ICI medication. This is of concern since weakness in the ascertainment of AEs in RCTs as well as non-prespecified decisions for determining the causality of AEs in protocols increase the risk of underreporting adverse drug reaction (ADRs), including irAEs.
Finally, the findings of this research bear significant implications for the conduction of future systematic reviews and meta-analyses of irAEs and irSAE from ICIs. The evaluation of all data sources including clinical trial registries and regulatory documents is not only recommended to limit the risk of publication bias but also for extracting results [13,14]. Although there have been numerous review articles on irAEs from ICIs, [15,38–41] none have evaluated and compared safety results pertaining to irAEs from publications, clinical trial registries and regulatory documents so far. While ClinicalTrials.gov provides specific guidelines for the reporting of certain safety results such as SAEs for investigators, [42] it does not have a minimum set of requirements or outline a standardized reporting format for irAEs / irSAEs, making the extraction of this information difficult. Consequently, the inconsistencies in the reporting formats of irAEs / irSAEs remain an impediment to the incorporation of relevant safety data from all existing sources in systematic reviews and meta-analyses, which compromises the quality of the overall evidence on irAE and irSAE.
This study had some limitations. First, because of the inconsistency in irAE reporting format, we were unable to provide a robust comparative assessment of irSAE reporting from all currently published ICI trials (only 5.9% had a comparable format). Second, we did not consider regulatory documents from the FDA in our assessment (such as drug package inserts and review documents) since only phase III published trials were included for this study’s purpose. Given that many metastatic cancers are considered as terminal and / or rare diseases, new drug approvals by regulatory authorities may be granted following phase II trials (pivotal trials) [43], which were excluded in the selection process of our study from the PubMed search results.
This study highlights the extensive variability in the reporting of safety information, particularly, irSAEs in RCTs evaluating ICIs. Adopting a standardized terminology and consistency in the reporting methods of safety information in published trials and clinical trial registries is imperative, not only to transparent communication in medical practice, but also for the incorporation of data from all existing sources in systematic reviews and meta-analyses.
AE – Adverse event
CTCAE – Common Terminology Criteria for Adverse Events
ICH – International Conference on Harmonisation
ICI – Immune-checkpoint inhibitor
irAE – Immune-related adverse event
irSAE – Serious immune-related adverse event
MedDRA – Medical Dictionary for Regulatory Activities
NCI – National Cancer Institute
PT – Preferred term
RCT – Randomized controlled trial
SAE – Serious adverse event
SOC – System organ class
trAE – Treatment-related adverse event
trSAE – Serious treatment-related adverse event
US-FDA – United States Food and Drug Administration
Funding – No external funding received
Conflicts of interest/Competing interests – The authors declare that they do not have any competing or conflict of interest that could have influenced the work reported in this paper.
Availability of data and material – The datasets used and/or analyzed in this study are available from the corresponding author upon reasonable request.
Ethics approval – Not applicable
Consent to participate – Not applicable
Consent for publication – All named authors have read and approved the final draft of the manuscript and agreed to its submission for publication. The authors confirm that the contents of this manuscript have not been published or submitted for publication elsewhere.
Code availability – Not applicable
Authors' contributions –
Acknowledgements
Any individual or organization that significantly contributed to this work has been acknowledged.
Published trials |
N=42 1 |
Type of journal Oncology General medicine |
14 (33.3%) 28 (66.7%) |
NCT number reported |
42 (100%) |
Immune-Checkpoint Inhibitors 2 Atezolizumab (Tecentriq®) Avelumab (Bavencio®) Cemiplimab (Libtayo®) Durvalumab (Imfinzi®) Ipilimumab (Yervoy®) Nivolumab (Opdivo®) Pembrolizumab (Keytruda®) |
5 (11.9%) 3 (7.1%) 0 (0%) 2 (4.8%) 13 (31.0%) 22 (33.3%) 10 (23.8%) |
ICI regimen Monotherapy with ICI Combination regimen of ICI with chemotherapy, radiotherapy, etc. |
26 (61.9%) 16 (38.1%) |
Medical indication Metastatic non-small cell lung cancer (NSCLC) Unresectable or metastatic melanoma Renal cell carcinoma (RC) Gastroesophageal / gastric cancer (GEC/GC) Head and neck squamous cell carcinoma (HNSCC) Urothelial carcinoma (UC) Prostate cancer (PC) Breast cancer (BC) Small cell lung cancer (SCLC) |
16 (38.1%) 11 (26.2%) 4 (9.5%) 3 (7.1%) 2 (4.8%) 2 (4.8%) 2 (4.8%) 1 (2.4%) 1 (2.4%) |
Study design Parallel with 2 arms Parallel with 3 arms |
36 (85.7%) 6 (14.3%) |
Blinding Open-label Double-blinded |
23 (54.8%) 19 (45.2 %) |
Primary outcomes Overall survival (OS) Progression Free Survival (PFS) Overall survival (OS) + Progression Free Survival (PFS) Recurrence Free Survival Safety outcomes Other (e.g., objective response rate, safety or other combinations) |
20 (47.6%) 1 (2.4%) 14 (33.3%) 3 (7.1%) 2 (4.8%) 2 (4.8%) |
RCT sites / countries At least one site in the USA No site in the USA |
37 (88.1%) 5 (11.9%) |
Other (European Organization for Research and Treatment of Cancer) |
42 (100%) 1 (2.4%) |
1. n (%), except otherwise indicated
2. The total percentages combined are more than 100% since 5 trials included both Ipilimumab and Nivolumab in one or more of their treatment arms
Formatting component |
Published trials |
ClinicalTrials.gov |
Causality Establishing drug-causality for AEs |
Primarily report treatment-related AEs (trAEs) |
All-cause AEs are reported regardless of drug causality |
Structural hierarchy The level at which different types of AE are reported |
System Organ Class (SOC) and / or Preferred Terms (PTs) according to MedDRA are used e.g., SOC: higher level group term (e.g., skin, GI) PT: lower level group term (e.g., rash, colitis) |
Report AE occurrence using PTs, but typically not by SOCs
|
Severity or grade The intensity of an AE (mild, moderate, severe, etc.) |
Often report AE grades; choice of presentation grading categories varies. e.g., some publications report grade 3-4 combined, others report grades 3, 4 and 5 together. |
Grading is most often not reported in trial registry results
|
Incidence of various types of AE Reporting the number of patients or events |
Generally report the number of AEs e.g., number of events which included rashes (including all grades and multiple episodes in patients, unless explicitly indicated that the highest grade per patient is reported) |
Usually report the number of patients who experienced each specific type of AE e.g., number of patients in treatment arm 1 who experienced serious autoimmune colitis |
Frequency threshold The incidence of AEs occurring beyond a certain threshold |
Authors often choose a higher frequency threshold to report AEs in the main text, however, they may choose to report a more comprehensive list using a lower cutoff in the supplementary tables e.g., 10% in the main table and 1% in the supplement |
ClinicalTrials.gov requires investigators to report all SAEs, and events ≥ 5% for non-SAEs |