New Benchmarks on Protocol Amendment Practices, Trends and their Impact on Clinical Trial Performance

doi:10.21203/rs.3.rs-3168679/v1

Download PDF

Research Article

New Benchmarks on Protocol Amendment Practices, Trends and their Impact on Clinical Trial Performance

https://doi.org/10.21203/rs.3.rs-3168679/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 04 Mar, 2024

Read the published version in Therapeutic Innovation & Regulatory Science →

You are reading this latest preprint version

The Tufts Center for the Study of Drug Development (Tufts CSDD) conducted a follow-up study in 2022 to assess trends in protocol amendment experiences and the impact amendments have had on clinical trial performance, particularly during the COVID-19 pandemic. Sixteen pharmaceutical companies and contract research organizations provided data on 950 protocols and 2,188 amendments. The results show that, since 2015, the prevalence of amendments in phase I – IV protocols has increased substantially (from 57–76%) and the mean number of amendments per protocol has increased 60% to 3.3, up from 2.1. Phase I and III protocols saw the highest increases in the mean number of amendments implemented per protocol. A much higher percentage of amendments — 77% —were deemed unavoidable with regulatory agency requests and changes to the study strategy as the top reasons cited for amending a protocol. In addition, the total average time to implement an amendment has nearly tripled during the past decade. The time from identifying the need-to-amend to last oversight approval now takes an average of 260 days and the mean duration during which investigative sites operate with different versions of the clinical trial protocol spans 215 days. Protocols that implemented at least one amendment were more effective at increasing patient screening volume and reducing the actual number of patients enrolled relative to plan. Lastly, the prevalence and mean number of amendments was significantly higher for protocols conducted during the pandemic.

Protocol amendments

protocol design

protocol complexity

protocol design changes

Although the implementation of protocol amendments are necessary for a variety of reasons (e.g., to protect study volunteer safety; improve patient recruitment and retention; and enhance study strategies) they are also highly disruptive and represent the largest single cause of unplanned delays and unbudgeted costs in clinical trials. Regardless of their extensive internal review, refinement and approval processes, most finalized protocols are amended multiple times resulting in months of delay and several hundred thousand dollars ($US) in direct costs. This includes adaptive protocol designs that must be amended despite pre-planned and pre-approved interim changes to the study design.

A 2010 study conducted by the Tufts Center for the Study of Drug Development (Tufts CSDD) found that the frequency of protocol amendments varied widely by therapeutic area. In that study, 17 large and mid-sized pharmaceutical companies provided data on 3,413protocols and 3,596 amendments. Nearly 60% of all protocols had at least one amendment with a mean of 2.6 and 3.6 total amendments per phase II and III protocols respectively. The 2010 study also found that the top reason for amending a protocol was to modify study volunteer eligibility criteria due to changes in study design strategy and difficulties recruiting patients and that the prevalence of amendments was highly associated with more complex protocols that have a higher relative number of endpoints, procedures, and investigative sites located in more countries. Finally, this study estimated that one-third of all protocol amendments could have been avoided.

In a follow-up 2015 study, 15 pharmaceutical companies and contract research organizations (CROs) provided data on 836 protocols and 984 amendments. The primary aims of this study were to update the 2010 study findings and to measure the impact of protocol amendments on clinical trial performance, cost and efficiency. Tufts CSDD observed a similar prevalence of protocols with at least one amendment (57%) though a much higher percentage of these amendments (45%) were deemed avoidable. A positive association was again observed between amendment prevalence and protocol complexity. This study also found that protocols with amendments were associated with longer timelines for study initiation, study execution, and study close-out. Protocols with amendments, however, resulted in fewer actual screened and enrolled patients relative to baseline plan than protocols without amendments.

In 2022, Tufts CSDD, in collaboration with 19 pharmaceutical companies and CROs conducted a follow-up study to continue assessing trends in protocol amendment experiences, add to our understanding of their impact and identify new opportunities to anticipate and prevent avoidable protocol amendments. This new study also offered insight into changes in amendment experience due to the global COVID-19 pandemic. The authors hope that these results will encourage discussion and inform new strategies to optimize protocol design and execution.

In April 2022, Tufts CSDD kicked-off this most recent study with representatives from the following sponsors and CROs: Amgen, AstraZeneca, Biogen, Bristol Myers Squibb, EMD Serono, Eli Lilly & Company, Gilead Sciences, IQVIA, Janssen Pharmaceutical Companies of Johnson & Johnson, Jazz Pharmaceuticals, Merck, Parexel, Pfizer, Regeneron, Roche/Genentech, Sanofi, Seagen, and Veristat. Working collaboratively with participating companies, Tufts CSDD determined the study sampling frame, and selected and defined study variables of interest based, in part, on past Tufts CSDD studies. Enhancements were made to the study methodology including measuring protocol writing cycle time; determining whether an amendment halted the study or paused recruitment during implementation; gathering more granular data on oversight of an amendment at the site level (i.e., date of first and last site oversight approval of an amendment); and assessing the impact of COVID-19 on amendment experience.

The data collection process was initiated in June 2022. Participating companies were asked to provide data on a representative sample of up to 100 randomly selected protocols -- with and without amendments. For those protocols with amendments, participating companies provided additional data about the nature, causes and impact of those amendments. A unique protocol identifier (NCT Number, protocol number, or other identifier, provided by the companies) was used to associate amendment data with the correct protocols.

The sampling requirements for protocols included and evaluated in this study were as follows:

Phase I – IV protocols, including pharmacology studies;
Excluding protocols for devices;
Including Traditional, adaptive and master protocol designs;
Achieved database lock (DBL) or primary completion date (PCD) between 2016 and 2021, with no restrictions on protocol approval year;
Randomly selected regardless of the presence of amendments, to be representative of company portfolios;
CROs should provide data on protocols from companies not already participating in the study.

Data were collected on protocol-specific characteristics including number of endpoints, procedures, countries, investigative sites engaged and patients enrolled; screening, recruitment and enrollment completion rates; planned and actual timelines and budgets. Data were also collected on substantial and country-specific amendments. Substantial amendments were defined as any change to a protocol on a global level requiring internal approval followed by approval by a regulatory authority and oversight body (e.g., ethical review committee); country-specific amendments included any change to the protocol that did not apply to all global locations. More granular data were only collected on substantial amendments and included: milestone dates of amendment implementation (e.g., internal approval, date of first oversight approval, etc.), primary and auxiliary amendment causes, specific changes made in response to an amendment, and direct and indirect amendment costs. Amendment causes included:

New Data Available (Other Than Safety Data)
New Safety Data Available
Change in Standard of Care
Regulatory Agency Request
Recruitment Difficulty
Manufacturing Change
Investigator / Site Feedback
Change in Study Strategy
Protocol Design Flaw
Inconsistency and / or Other Error in Protocol
COVID-19 Pandemic

Protocol design areas that were changed in response to implementing an amendment included General Information; Trial Background; Objectives and Purpose; Trial Design; Selection, Withdrawal, and Treatment; Assessments; Statistics; Operational; and Other.

All participating companies collected and coded their own data. Questions about variable definitions and data collection methods were handled by the Tufts CSDD team. Data collection was completed in October 2022. All collected data were reviewed and cleaned by the Tufts CSDD team. Numerical errors and incomplete data elements were returned to participating companies for clarification and correction.

Data were merged and analyzed using SAS 9.4. Any protocols falling outside of the sampling requirements were removed from the analysis dataset. Descriptive statistics were calculated and chi-square and analyses of variance (ANOVA) were conducted to test for significant differences in the prevalence and mean number of amendments by clinical trial characteristics including phase, therapeutic area, use of DCT methodologies, and New Molecular Entity (NME) Type (large molecule, small molecule, or vaccine). Mean, coefficient of variation, median, and range were calculated for cycle times. Amendment impact was measured by comparing the difference between planned and actual performance outcomes including cycle times (in days), protocol budget, scope, and enrollment for protocols with and without amendments. Change in enrollment performance—including Screen Failure (total enrolled / total screened) and Completion Rate (total enrolled / total completed)—was also calculated based on planned and actual raw enrollment numbers for protocols with and without amendments. Actual Screen Failure, Completion, and Dropout Rate (total dropouts / total enrolled) were calculated for protocols with amendments before (pre-pandemic) and during the pandemic. These changes were measured as a percent change from plan to account for differences in protocol size.

Causes of amendments, and the associated changes made during implementation, were assessed overall; for oncology compared to non-oncology clinical trials; for clinical trials conducted pre- and during the COVID-19 pandemic; and by key milestones in the timing of amendment implementation: before First Patient First Visit (FPFV), between FPFV and Last Patient First Visit (LPFV), and after Last Patient First Visit (LPFV)).

Similar to past Tufts CSDD studies, participating companies classified amendments into four categories based on primary cause: Completely Avoidable (e.g., Protocol Design Flaw; Inconsistency and / or Other Error in Protocol), Somewhat Avoidable (e.g., Recruitment Difficulty; Investigator/Site Feedback), Somewhat Unavoidable (e.g., New Data Available (Other than Safety Data); Change in Standard of Care; Change in Study Strategy), and Completely Unavoidable (e.g., New Safety Data Available; Regulatory Agency Request; Manufacturing Change; due to the COVID-19 Pandemic).

Sixteen companies provided data on 950 qualifying protocols with 2,188 qualifying amendments. Completion rates of individual variables in our sample were generally high; 87% of protocols contained general characteristics and 72% had data on study scope, time and budget. Protocol design variables had lower relative completion rates of approximately 50%. Protocol amendment timeline and impact variables had an average completion rate just below 40%. Amendment causes and changes had average completion rates of 57%. Cost data proved extremely difficult for participating companies to provide, with direct cost variables having a very low completion rate of only 1%; no participating companies were able to provide indirect cost data. As such, we are unable to analyze and report on amendment cost benchmarks or trends.

[Insert Table 1]

Table 1

Protocol Sample and Amendment Sample Characteristics
Characteristic		Protocol Sample		Amendment Sample
Characteristic		n	Percent	n	Percent
Phase
	Phase I	358	37.7%	685	31.3%
	Phase II	251	26.4%	686	31.4%
	Phase III	286	30.1%	757	34.6%
	Phase IV	55	5.8%	60	2.7%
NME Type
	Large Molecule	358	44.8%	1,058	53.1%
	Small Molecule	386	48.3%	825	41.4%
	Vaccine	56	7.0%	111	5.6%
Orphan Designation
	Yes	41	8.3%	106	9.4%
	No	451	91.7%	1,021	90.6%
Therapeutic Area
	Oncology	249	26.2%	779	35.6%
	Immunology / Rheumatology	152	16.0%	378	17.3%
	Neurology	150	15.8%	226	10.3%
	Infectious Disease	107	11.3%	216	9.9%
	Gastroenterology / Endocrinology	70	7.4%	130	5.9%
	Vaccines	55	5.8%	109	5.0%
	Dermatology	40	4.2%	63	2.9%
	Hematology	39	4.1%	110	5.0%
	Cardiology	30	3.2%	77	3.5%
	Respiratory	20	2.1%	28	1.3%
	Musculoskeletal	4	0.4%	10	0.5%
	Orthopedics	1	0.1%	0	0.0%
	Other	33	3.5%	62	2.8%
Participant Population
	Patients	657	72.1%	1,741	83.6%
	Healthy Volunteers	238	26.1%	310	14.9%
	Both	16	1.8%	31	1.5%
Participant Age
	Adult (18+)	795	89.7%	1,908	89.2%
	Pediatric (< 18)	59	6.7%	171	8.0%
	All Ages	32	3.6%	59	2.8%
Company Type
	Sponsor	915	96.3%	2,096	96.9%
	CRO	35	3.7%	67	3.1%
*Amendment sample includes multiple amendments per protocol

Table 1 presents the sample size and distribution of protocols and amendments analyzed. Overall, our sample included a strong mix of protocols and amendments by phase, therapeutic area and molecule size.

Table 2 presents the prevalence of amendments and the mean number of amendments implemented where amendments were present. Overall, more than three quarters of protocols required at least one amendment, with a mean of 3.3 amendments implemented per protocol when required. Protocols for large molecules had a higher prevalence of amendments and required a higher mean number to be implemented than did protocols for small molecules or vaccines. Oncology protocols had a higher prevalence and a higher mean number of amendments at 90% and 4.0 respectively. Protocols targeting diseases in immunology / rheumatology and protocols testing vaccines showed the next highest prevalence of amendments (79% and 75%, respectively), however they required implementing a smaller mean number of amendments than the overall average at 3.2 and 2.7, respectively.

Table 2

Prevalence and Number of Amendments
Protocol Characteristic		n	Percent with Amendments	When Amendments are Present
Protocol Characteristic		n	Percent with Amendments	n	Mean (CoV)	Median
Overall		864	75.8%	690	3.3 (0.74)	3.0
Phase
	Phase I	326	65.3%	228	3.1 (0.74)	3.0
	Phase II	221	88.2%	212	3.3 (0.71)	3.0
	Phase III	269	82.2%	224	3.5 (0.76)	3.0
	Phase IV	48	54.2%	26	2.4 (0.59)	2.0
NME Type
	Large Molecule	346	81.5%	294	3.7 (0.69)	3.0
	Small Molecule	362	70.2%	277	3.1 (0.78)	2.0
	Vaccine	56	75.0%	42	2.6 (0.69)	2.0
Orphan Designation
	Yes	41	70.7%	29	3.7 (0.74)	3.0
	No	451	81.6%	368	2.8 (0.61)	3.0
Therapeutic Area
	Oncology	198	89.9%	205	4.0 (0.67)	3.0
	Immunology / Rheumatology	148	79.1%	121	3.2 (0.79)	3.0
	Vaccines	55	74.6%	41	2.7 (0.70)	2.0
	Infectious Disease	106	74.5%	80	2.9 (0.63)	3.0
	Hematology	39	74.4%	29	3.9 (0.77)	3.0
	Cardiology	30	70.0%	21	3.6 (0.64)	3.0
	Neurology	121	69.4%	87	2.6 (0.60)	2.0
	Dermatology	40	65.0%	26	2.1 (0.54)	2.0
	Respiratory	20	65.0%	13	2.1 (0.42)	2.0
	Gastroenterology / Endocrinology	69	62.3%	43	3.1 (0.97)	2.0
	Musculoskeletal	4	50.0%	2	5.0 (0.28)	5.0
	Orthopedics	1	0.0%	0	- (-)	-
	Other	33	66.7%	22	3.5 (0.86)	3.0

Although two-thirds of implemented amendments resulted in the need to reconsent study volunteers, less than 2% resulted in pausing recruitment or halting the clinical trial. And, with respect to amendments implemented in a single or partial number of geographic areas, phase II and III clinical trials had the highest proportion of country-specific amendments at 44.8% and 60.1% of all protocols respectively.

[Insert Tables 2 and 3]

Table 3

Amendment Prevalence and Frequency Trends
	2010 Study		2015 Study		2022 Study
Protocol Characteristic	% Protocols w/ Amendments	Mean Amendments, When Amendments are Present	% Protocols w/ Amendments	Mean Amendments, When Amendments are Present	% Protocols w/ Amendments	Mean Amendments, When Amendments are Present
Overall	69%	2.4	57%	2.1	76%	3.3
Phase I		2.0	52%	1.8	65%	3.1
Phase II		2.6	77%	2.2	88%	3.3
Phase III		3.6	66%	2.3	82%	3.5
Phase IV		2.3	25%	1.9	54%	2.4
Gastrointestinal		4.4		2.3	100%	3.5
Cardiovascular		3.4		1.9	70%	3.6
Oncology		3.2		2.6	90%	4.0
Respiratory		3.2		1.3	65%	2.1
Hematology		2.3		1.5	74%	3.9
Immunology / Inflammation		1.8		2.1	76%	3.1

While the overall prevalence and mean number of amendments per protocol decreased between the 2010 and 2015 studies, the prevalence and mean number of amendments implemented per protocol both increased considerably between 2015 and the 2022 time period (refer to Table 3). Amendment prevalence has increased in each phase since 2015, as has the average number of amendments implemented per protocol when amendments were present. Compared to 2010, the average number of amendments implemented when amendments were present also increased for every phase except Phase III, where the average decreased modestly (3.6 in 2010 and 3.5 in 2022). The average number of amendments has increased since 2015 in all therapeutic areas measured in our study with the exceptions of gastrointestinal (4.4 in 2010 and 3.5 in 2022) and respiratory (3.2 in 2010 and 2.1 in 2022) disease protocols.

[Insert Table 4]

Table 4

Trends in Amendment Timing
	Before FPFV			During Enrollment (FPFV – LPFV)		During Study Maintenance (After LPFV)
Phase	2010 Study	2015 Study	2022 Study	2015 Study	2022 Study	2015 Study	2022 Study
Overall	43%	23%	24%	62%	58%	15%	18%
Phase I	52%	40%	25%	50%	63%	10%	13%
Phase II	37%	18%	26%	70%	54%	12%	19%
Phase III	30%	15%	22%	65%	57%	20%	21%
Phase IV	38%	33%	28%	67%	56%	0%	16%

Table 4 examines trends in the timing of amendment implementation. Overall, the percentage of amendments occurring before FPFV has decreased since 2010, but there has been minimal change between 2015 and 2022. Several trends are, however, observed in the timing of amendment implementation by phase. Among Phase I protocols there has been a steady decrease in the percentage of amendments occurring before FPFV from 52% in 2010, to 40% in 2015, down to 25% in 2022. A rising trend is observed in the percentage of amendments occurring during enrollment from 50% in 2015 to 63% in 2022. In both Phase II and Phase III protocols, the percentage of amendments occurring before FPFV decreased between 2010 and 2015 but increased between 2015 and 2022, with both phases showing a decrease in the percentage of amendments occurring during enrollment between 2015 and 2022.

The reported percentage of unavoidable amendments – including those deemed ‘Somewhat Unavoidable’ and ‘Completely Unavoidable’ – has increased substantially since the 2010 and 2015 Tufts CSDD studies (refer to Table 5). Table 5 also shows that changes in clinical trial strategy and regulatory agency requests are consistently among the most common primary causes of amendments in 2022. Protocol design flaws and recruitment difficulties were among the top five reported reasons for amending protocols in the earlier 2010 and 2015 Tufts CSDD studies.

Table 5

Trends in Amendment Causes and Avoidability
Avoidability	2010 Study	2015 Study	2022 Study
Completely Avoidable	18%	23%	17%
Somewhat Avoidable	19%	22%	6%
Somewhat Unavoidable	27%	30%	37%
Completely Unavoidable	39%	25%	40%
Top 5 Reasons for Amendments	New safety information available	Protocol design change / clarification	Regulatory agency request
	Regulatory agency request	Administrative	Change in study strategy
	Change in strategy / objective	Change to analysis	New data available (other than safety)
	Protocol design flaw	Recruitment difficulty	Inconsistency / error in protocol
	Recruitment difficulty	Feedback from sites	New safety data available

[Insert Table 5]

Amendment implementation, measured from internal approval of the amendment to the last required approval by an ethical review committee or oversight body takes — on average — approximately 260 days, with a median of 190 days (refer to Table 6). The duration from internal approval of an amendment to first patient reconsented is 89 days in the 2022 study, more than 2.5 times longer than the duration observed in our 2010 study. During amendment implementation, the duration of time in which sites are operating with different versions of a clinical trial protocol (i.e., time between the first required approval by an ethical review committee to the last required approval by an ethical review committee) was just over 7 months (215 days).

Table 6

Amendment Implementation Cycle Times
Cycle Time (in Days)	n	Mean (CoV)	Median	Range
Amendment Implementation (Internal Approval to Last Oversight Approval)	391	259.7 (0.95)	190.0	1–1,599
Submission Preparation (Internal Approval to First Oversight Submission)	352	37.8 (3.06)	14.0	0–1,147
Time to First Approval (Internal Approval to First Oversight Approval)	372	61.9 (1.97)	37.0	0–1,177
Time to Reconsent (Internal Approval to First Patient Reconsented)	71	88.8 (1.36)	52.0	0–666
Duration of Sites on Different Protocol Versions (First Oversight Approval to Last Oversight Approval)	393	214.8 (1.15)	140.0	0–1,583

[Insert Table 6 ]

Table 7 presents mean actual cycle times for primary clinical trial time points for protocols with and without amendments and shows trends in these cycle times since 2015. Protocols with amendments generally have longer cycle times than do those without amendments. Protocols with at least one amendment are associated with larger and more complex studies having nearly 25% more endpoints and 16% more eligibility criteria than do those without an amendment. The average number of countries and investigative sites per protocol were also significantly greater for protocols with amendments. Protocols without at least one amendment screen and enroll 46% and 37% fewer patients, respectively.

Table 7

Cycle Times for Protocols with and without Amendments
Cycle Time		Without Amendments		With Amendments		p-value
Cycle Time		n	Mean	n	Mean	p-value
Study Initiation (Protocol Approval – FPFV)	2015 Study	12	154	14	181	.45
Study Initiation (Protocol Approval – FPFV)	2022 Study	198	176	666	181	.7457
Study Conduct (Protocol Approval – LPLV)	2015 Study	235	490	477	580	.0003
Study Conduct (Protocol Approval – LPLV)	2022 Study	200	546	617	1,121	< .0001
Study Closeout (LPLV – DBL)	2015 Study	33	140	127	230	.007
Study Closeout (LPLV – DBL)	2022 Study	195	58	564	49	.3868
First Visit Duration (FPFV – LPFV)	2015 Study	215	238	286	403	< .0001
First Visit Duration (FPFV – LPFV)	2022 Study	190	234	625	516	< .0001
Enrollment Completion (FPFV – LPLV)	2015 Study	225	352	439	437	< .0001
Enrollment Completion (FPFV – LPLV)	2022 Study	195	370	617	942	< .0001

With respect to cycle time trends, study initiation durations did not change significantly since 2015. Study conduct, first visit duration, and enrollment completion cycle times all increased significantly. Moreover, the difference in cycle time lengths between protocols with and without amendments has also widened since 2015.

[Insert Table 7]

Enrollment performance in clinical trials without protocol amendments tends to perform closer to plan than do those that implement at least one amendment. The actual number of patients screened, enrolled and completing protocols without amendments was consistently closer to plan or forecast (see Table 8). The difference between plan and actual durations overall and for individual time points are also substantially longer for protocols with at least one amendment. Protocols with at least one amendment require screening more patients and have much higher screen failure rates than initially planned. However, they recruit and enroll far fewer patients than originally planned.

Table 8

Percent Change in Enrollment and Cycle Times for Protocols with and without Amendments
% Change in –	No Amendments			With Amendments
% Change in –	n	Mean (CoV)	Interpretation	n	Mean (CoV)	Interpretation
Participants Screened	60	0.7% (95.40)	Actual > Planned	339	8.6% (7.35)	Actual > Planned
Participants Enrolled	176	-6.1% (5.97)	Actual < Planned	620	-7.8% (4.90)	Actual < Planned
Participants Completed	56	-3.9% (9.89)	Actual < Planned	329	-20.8% (2.56)	Actual < Planned
Screen Failure Rate	46	-6.9% (11.84)	Actual < Planned	297	31.0% (9.73)	Actual > Planned
Completion Rate	54	-0.7% (45.12)	Actual < Planned	327	-19.3% (2.48)	Actual < Planned
Study Startup (Approval to FPFV)	126	2.8% (10.17)	Actual > Planned	593	18.2% (4.43)	Actual > Planned
First Visit Cycle (FPFV to LPFV)	114	2.2% (28.95)	Actual > Planned	529	14.1% (5.84)	Actual > Planned
LP Participation Cycle (LPFV to LPLV)	110	45.9% (8.56)	Actual > Planned	504	34.8% (6.24)	Actual > Planned
Closeout (LPLV to DBL)	72	27.6% (7.99)	Actual > Planned	440	59.0% (4.01)	Actual > Planned
Study Reporting (Database Lock to CSR)	69	34.0% (4.33)	Actual > Planned	433	32.5% (8.43)	Actual > Planned
Enrollment Duration (FPFV – LPLV)	123	2.4% (14.01)	Actual > Planned	567	14.0% (5.66)	Actual > Planned
Total Study Duration (Approval to DBL)	77	3.8% (6.94)	Actual > Planned	463	13.9% (3.26)	Actual > Planned

[Insert Table 8]

Protocols that completed their last patient’s last visit by March 1st, 2020 were considered to have occurred before the onset of the COVID-19 pandemic. Those that completed LPLV after that date screened, recruited and enrolled patients during the COVID-19 pandemic. Table 9 presents comparisons between plan and actual scope and patient recruitment and retention rates for protocols conducted before and during the pandemic. The results show significantly higher relative increases in the number of endpoints and countries added to protocols requiring amendments during the pandemic. The mean number of amendments per protocol increased significantly during the pandemic, from 3.0 to 3.5 (p = .0063). A significant decrease in screen failure rates (34.8–31.3%, p = .0362) and completion rates (71.5–65.0%, p = .0193), and a significant increase in dropout rates (18.8–29.6%, p < .0001) are associated with protocols that were amended during the pandemic.

Table 9

Comparing Protocols Before and During the Pandemic
	Pre-Pandemic			Pandemic
	n	Mean (CoV)	Median	n	Mean (CoV)	Median	p-value
Amendments*	302	3.0 (0.74)	2.0	319	3.5 (0.70)	3.0	.0063
Screen Failure Rate	383	34.8% (0.69)	32.5%	380	31.3% (0.72)	28.6%	.0362
Completion Rate	350	71.5% (0.49)	88.0%	323	65.0% (0.56)	79.6%	.0193
Dropout Rate	313	18.8% (1.37)	8.9%	311	29.6% (1.14)	15.4%	< .0001
Change in Scope* (Planned vs Actual)
Countries	251	14.5% (9.70)	0.0%	259	56.7% (5.38)	0.0%	.0448
Sites	275	-1.6% (24.81)	0.0%	296	-7.4% (7.86)	-2.3%	.1669
Endpoints	202	7.7% (4.24)	0.0%	218	17.5% (3.10)	0.0%	.0245
Eligibility Criteria	203	3.9% (5.15)	0.0%	221	6.0% (5.55)	0.0%	.4220
*When amendments are present

Despite high industry-wide awareness and recognition of the unplanned time and unbudgeted cost associated with implementing protocol amendments, the prevalence and mean number of amendments per protocol have increased across all clinical phases since 2015. Overall, the average number of amendments per protocol in 2022 increased nearly 60% between 2015 and 2022. More specifically, phase I and III protocols saw the highest increases in the mean number of amendments implemented per protocol over this seven-year period. This study found that more than 75% of protocols require at least one substantial amendment with the highest prevalence (89%) observed in phase II protocols. And clinical teams now implement an average of 3.3 amendments per protocol with the highest mean number (3.5) in Phase III clinical trials. Compared to protocols without amendments, those with at least one amendment were larger in scope and took significantly longer overall as well as at each critical clinical trial segment (e.g., initiation, conduct, close out).

The upward trend in the prevalence and frequency of amendments may be explained, in part, by the strong positive relationship between protocol complexity (e.g., higher relative number of endpoints, eligibility criteria, procedures, countries and investigative sites) and amendments and the steady increase in the scientific and operating complexity of protocol designs observed by Tufts CSDD over the past decade. The most notable factors driving complexity at this time include the growing number of drugs in active clinical trials targeting rare diseases, more narrowly defined patient populations and more logistically demanding and complicated clinical trial execution models involving more countries, investigative sites, technology solutions and primary data sources (e.g., mobile and wearable devices; specialized assessments)^, .

Although the gap between actual durations and those initially planned were wider for protocols with amendments than for those without, protocols that implemented at least one amendment were more effective at increasing patient screening volume and reducing the actual number of patients enrolled relative to plan.

A much higher percentage of amendments (77%) were deemed unavoidable, up from 66% in the 2010 Tufts CSDD study. This finding may reflect changing sponsor perceptions that a growing majority of amendments are necessary. This trend corresponds with the increase in the percentage of sponsor companies reporting that regulatory agency requests and changes in study strategy were top reasons for amending a protocol. Intensifying competition from a larger community of active sponsors during drug development through commercialization, increasing formal and informal sponsor-regulatory agency interactions, and growing agency demand for patient-centric clinical trial designs and executional models may help explain the prominence of these reasons at the present time for amending protocols.

The results of this study characterize the stunning toll that protocol amendment implementation takes on clinical trial timelines and the potential for high levels of uncertainty between investigative sites administering the protocol. The total average time to implement an amendment for example — from identifying the need-to-amend to last oversight approval — was a remarkable 260 days. During implementation, the average duration during which investigative sites were operating with potentially different versions of the clinical trial protocol was 215 days. And the results showed a 154% increase since 2010 in the time between internal approval-to-implement-an- amendment and re-consent of the first patient.

We were not surprised to observe an increase in the prevalence and mean number of amendments and the corresponding increase in the number of endpoints and countries during the pandemic. The unique challenges of conducting clinical trials during the pandemic — including the extra precautions taken and social distancing required, heightened levels of remote interaction and concerns about transmitting the COVID-19 virus — all contributed to greater uncertainty and anxiety among study volunteers and study staff. Indeed the results indicated that protocols conducted during the pandemic had significantly higher study volunteer dropout rates.

In the Tufts CSDD 2010 and 2015 studies we advocated for protocol simplification and a focus on preventing avoidable amendments — particularly those implemented before study volunteers received the first dose of the investigational drug — as primary strategies to minimize the highly disruptive nature and delays associated with substantial amendments. The results of this recent study suggest that these strategies may be only marginally effective. Protocol design complexity has been rising and the proportion of avoidable amendments has been declining since 2015.

A separate but recent Tufts CSDD study, however, found that soliciting input into draft protocol designs from patients and investigative sites did improve clinical trial performance and reduced the number of protocol amendments. Notable differences were observed in protocols informed by a patient advisory board including faster study initiation, clinical trial and study close-out durations, lower-than-planned study budgets, and a significantly smaller average number of substantial protocol amendments.

Results from the 2022 study also point to the importance of strategies that hold greater potential to drive more efficient and shorter protocol amendment implementation timelines. Technology solutions that improve communication and coordination between internal teams and collaborators (e.g., CROs, global investigative sites and oversight bodies) through faster and more efficient data-sharing will play an important role. Over time, automation and AI-enabled approaches may also help drive faster and more efficient implementation of amendments.

This study had limitations of note. The data collected came from a biased sample of primarily mid-sized and major pharmaceutical companies. However, we have no data that evidences that protocol amendment experience are different in clinical trials sponsored by small companies. Due to the timing of this study, we do not have results that characterize protocol amendment experience post pandemic. Tufts CSDD plans to conduct a follow-up study in the coming years to inform our understanding of experiences after the ‘official’ end of the global pandemic. We were also unable to collect data on the cost to implement amendments in part due to the difficulty that participating companies had in accessing data from other functions in their organization and from vendors. Tufts CSDD is currently planning to do a follow-up study focusing on gathering robust protocol amendment cost data.

The highly disruptive and unplanned impact of substantial protocol amendments on clinical trial timelines and costs combined with their growing prevalence and frequency point to a priority challenge, and an unprecedented opportunity, to drive drug development efficiency and speed.

Acknowledgments

Thank you to the many companies that participated and supported this working group study: Amgen, AstraZeneca, Biogen, Bristol Myers Squibb, EMD Serono, Eli Lilly & Company, Gilead Sciences, IQVIA, Janssen Pharmaceutical Companies of Johnson & Johnson, Jazz Pharmaceuticals, Merck, Parexel, Pfizer, Regeneron, Roche/Genentech, Sanofi, Seagen, and Veristat.

Conflict of Interest Statement

Kenneth Getz, Tufts CSDD, has nothing to disclose.

Zachary Smith, Tufts CSDD, has nothing to disclose.

Emily Botto, Tufts CSDD, has nothing to disclose.

Elisabeth Murphy, Janssen Pharmaceutical Companies of Johnson & Johnson, declares that she is an employee and has financial holdings in the company.

Arnaud Dauchy, Sanofi, declares that he is an employee and has financial holdings in the company.

Author Contributions:

Kenneth Getz, Tufts CSDD, contributed to all four aspects (substantial contribution to conception, design, analysis, interpretation; drafting and revising the work; final approval of the version to be published; agreement to be accountable for all aspects in ensuring accuracy and integrity of the work)

Zachary Smith, Tufts CSDD, contributed to all four aspects;

Emily Botto, Tufts CSDD, contributed to all four aspects;

Elisabeth Murphy, Janssen Pharmaceutical Companies of Johnson & Johnson, made substantial contribution to conception, design, analysis, interpretation.

Arnaud Dauchy, Sanofi, made substantial contribution to conception, design, analysis, interpretation.

Getz K, Stergiopoulos S, Short M et al. The impact of protocol amendments on clinical trial performance and cost. Therapeutic Innovation and Regulatory Science. 2016; 50(4): 436–441.
Ibid
Kaitin K. Cite Impact Report
Getz K, Zuckerman R, Cropp A, Hindle A, Krauss R, Kaitin K. Measuring the incidence, causes, and repercussions of protocol amendments. Drug Information Journal 2011; 45: 265 − 75.
Getz K, Stergiopoulos S, Short M et al. The impact of protocol amendments on clinical trial performance and cost. Therapeutic Innovation and Regulatory Science. 2016; 50(4): 436–441.
Getz K, Smith Z, Kravet M. Protocol design and performance benchmarks by phase and by oncology and rare disease subgroups. TIRS 2023; 57(1): 49–56.
Cite IQVIA Report
Cite Getz – new benchmarks paper
Cite IQVIA and/or Evaluate Pharma Report
Getz K. Amplifying patient voices in protocol design. Applied Clinical Trials, 2021; 30 (9): 10–12.

Download PDF

Journal Publication

published 04 Mar, 2024

Read the published version in Therapeutic Innovation & Regulatory Science →

Editorial decision: Major revisions
20 Oct, 2023
Reviewers agreed at journal
02 Aug, 2023
Reviewers invited by journal
25 Jul, 2023
Editor assigned by journal
17 Jul, 2023
First submitted to journal
13 Jul, 2023

You are reading this latest preprint version

New Benchmarks on Protocol Amendment Practices, Trends and their Impact on Clinical Trial Performance

Status:

Journal Publication

Version 1

Abstract

Introduction

Methods

Results

Discussion

Declarations

References

Status:

Journal Publication

Version 1