Improving Clinical Trial Design Using Interpretable Machine Learning Based Approach

doi:10.21203/rs.3.rs-2001465/v1

Download PDF

Article

Improving Clinical Trial Design Using Interpretable Machine Learning Based Approach

https://doi.org/10.21203/rs.3.rs-2001465/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 04 Jan, 2023

Read the published version in Scientific Reports →

You are reading this latest preprint version

This study proposes using a machine learning pipeline to optimise clinical trial design. The goal is to use machine learning modelling to predict early termination probability of clinical trials and to understand feature contributions driving this outcome to make further suggestions to the study protocol to reduce the risk of wasted resources. A dataset containing 420,268 clinical trial records and 24 fields was extracted from the ct.gov registry. In addition to study characteristics features, this study uses 12,864 eligibility criteria search features generated using a public annotated eligibility criteria dataset, CHIA. Ensemble models including random forest and extreme gradient boosting classifiers were used for training and evaluating predictive performance. We achieved a Receiver Operator Characteristic Area under the Curve score of 0.78, and balanced accuracy of 0.70 on the test set using xgBoost. We used Shapley Additive Explanations (SHAP) to interpret our black box machine learning models to make suggestions on trial protocol of any test instance. This pipeline will lead to an optimised clinical trial design and consequently will help potentially life-saving treatments reach patients faster.

Clinical research includes interventional and observational studies that use volunteers to experiment if a new drug, treatment or medical device is efficient and safe in a particular medical area. Clinical trials that fail prematurely due to poor design are a waste of resources and deprives society of data for evaluating potentially effective interventions. The cost of failed trials is larger in later phase trials, such as Phase 2 and Phase 3 trials, due to their larger sample sizes and investment in prior trials ¹. Hence, an optimised trial design is important to ensure a successful clinical study, well used resources and ethical use of participants.

Although AI/ML has been used in the clinical trials domain it is mostly focused on predicting trial outcomes rather than focusing on the design of clinical trials. Currently, trial design is largely manual. ClinicalTrials.gov contains over 400,000 clinical study records from 220 countries where studies are registered to the website by the sponsor or principal investigator of the clinical study ². Unlocking the potential of past data using machine learning, it is possible to reduce the time and cost spent on the trial design process and make recommendations to generate the most effective version of a study protocol.

Related Works

Clinical trial success can have two different definitions; the success of the intervention or successful completion of the trial (whether the intervention achieved its objective/s). For the scope of this study, we use the latter definition. Success is defined as the successful completion of a study.

There are several studies in the literature focused on predicting clinical trial success using machine learning algorithms based on structured and unstructured features collected from clinical studies. These studies focus on predicting early-termination of clinical studies using trial characteristic data combined with unstructured data. Follett et al. used structured and unstructured data together to predict clinical trial termination³. The study uses text mining to generate vector features from description fields of the studies in addition to basic study characteristics features. Selecting completed and terminated trials, a dataset of 130,000 studies was created from information that is available prior to the start of the study. They then use machine learning techniques to predict outcomes.

Research conducted by Elkin et al. uses machine learning to predict clinical trial termination from data extracted from clinicalTrials.gov ⁴. The study generates a dataset of 68,999 studies, 640 features are generated by using document embeddings, keyword features and trial characteristics. The purpose of their study was to understand clinical trial termination at a deeper level and achieve satisfactory prediction results.

Contribution

Our research has notable contributions to the existing research in this field. Firstly, in addition to using study characteristics features, we generate a set of new eligibility criteria category / entity pair features and use an n-gram model to search these binary features in the free-text eligibility criteria column for each study. Furthermore, we add disease category features based on the clinicaltrials.gov categorization where a study could belong to more than one of these categories. Finally, we have used SHAP (SHapley Additive exPlanations) to interpret individual predictions of our black box machine learning models. These explanations are used to make effective suggestions to the study protocol to minimise the risk of early termination.

Data source. The database for aggregate analysis of ClinicalTrials.gov (AACT) is a publicly available relational database enhanced by The Clinical Trials Transformation Initiative (CTTI) that contains both protocol and result elements for all studies recorded in ClinicalTrials.gov ⁵. The AACT database had 420,268 clinical studies registered from 1999 to July 2022 recorded in their database on 6 July 2022 and this version is extracted in comma-separated values (CSV) format from the database for this research.

Filtering the data. The dataset is filtered to only have “Interventional” studies out of the three study types included in the raw dataset are “Interventional”, “Observational” and “Patient Registry”. There are 14 different overall status records for this dataset including recruiting, completed, withdrawn and unknown status. Only studies that are completed, terminated or withdrawn are extracted from the overall dataset as these will be our focus point to generate our binary classification models.

The objective of clinical study registries is to provide complete, accurate and timely recorded trial data. Although the emphasis on registering clinical studies and providing quality data increases over time (since 1999) there are still a high number of studies that have a significant number of missing data points or have substantial errors ⁶. Figure 1 illustrates a bar plot which shows the average missing values ratios starting from 1999 where the first study registered to this registry until 2022 considering the 24 study design features investigated in this research. Average missing values are calculated as the proportion of sum of missing values over total number of studies recorded that year. There is a significant decrease in the average missing value rates over the years especially from 1999 to 2008 and after 2011 the average missing value rate is stable below 10. Hence, the studies registered before 2011 are removed from the dataset, leaving 112,647 studies.

Study characteristics features. Table 1 gives a summary of the number of studies and their recorded phases in the dataset. Some studies are recorded as belonging to two phases. For phase specific subset data generation, if the study phase recorded for a study includes the phase of interest, then it is included in that subset. For example, if a study phase is recorded as Phase2/Phase3 it will be included in both Phase 2 and Phase 3 subsets.

Out of 190,678 studies, 19,252 studies did not record the number of sites. Although the recent growth in decentralised trials emphasised that sites are not always needed, most of the historical studies do not belong to this category of trials ⁷. The number of clinical sites have an impact on the trial enrolment and patient demographics as the clinical study is limited by the participants who live near the defined sites and can attend study visits Therefore, we used the number of sites in the final features set.

One of the essential parts of the clinical trial design is to define the primary and secondary outcomes ⁸. The primary outcome measures directly form part of the study hypothesis. The number of primary and secondary outcomes to measure are included in our final feature set as two separate features.

Table 1 Study phases and number of studies recorded under each phase in clinicaltrials.gov

Phase	Number of Studies
Early Phase 1	1549
Phase 1	23914
Phase 1/Phase 2	5350
Phase 2	25599
Phase 2/Phase 3	2910
Phase 3	19699
Phase 4	16463

Figure 2 illustrates proportions of completed and failed studies by disease category. Categorization of a clinical study is done by searching through the conditions recorded for that study. This categorization of specific conditions is the same used in the clinicalTrials.gov database. As illustrated, studies under neoplasms and blood lymph conditions categories are the most likely to fail whereas studies under occupational diseases and disorders of environmental origin are the least likely to fail. Full list of study characteristics features is provided in supplementary materials.

Eligibility criteria statistical and search features. The eligibility criteria field is used to create a set of features including some basic descriptive features as well as more complex search features. Eligibility criteria are implemented to control who can participate in clinical studies. Acceptance of healthy volunteers, acceptance of patients by their genders and ages are among the features added followed by number of inclusion and exclusion criteria as well as total and average number of words for eligibility criteria per study. 54,758 studies that accepted healthy volunteers had a 7% failure rate whereas 134,842 studies that did not accept healthy volunteers had a 17% failure rate. The importance of inclusive eligibility criteria has been emphasised increasingly over the years as exclusion of particular subgroups makes it harder for studies to recruit patients and deliver inclusive outcomes ⁹.

The next set of features generated from the free-text eligibility criteria field are defined as search features. The CHIA dataset is a public dataset for a shared benchmark generated by Kury et al ¹⁰. It is a large, annotated corpus of patient eligibility criteria extracted from 1,000 Phase IV studies registered in ClinicalTrials.gov. The dataset contains 12,864 inclusion and exclusion criteria. We use CHIA to create search terms to extract features from the free-text inclusion and exclusion fields.

We use the following category types in CHIA to generate our search terms: "Condition", "Procedure", "Person", "Temporal", "Drug", "Observation", "Mood", "Visit". Category and entity pairs are generated for inclusion and exclusion criteria separately. 12,864 pair features were generated, and these pairs are used to search the inclusion and exclusion free text fields for features. For computational reasons, we restricted search terms to those with 5 or less words. This process generated sparse binary features which we concatenated to our original features.

Data labelling. Overall status is recorded for every clinical trial in the AACT database. If no participants were ever enrolled in the trial, the status of that trial is ‘Withdrawn’ and if a trial was stopped prematurely, the status of that trial is ‘Terminated’. Out of 28,098 terminated or withdrawn studies that reported a reason for stopping the study, 9,260 studies prematurely stopped due to reasons related to participant recruitment and enrolment. Trials that are successfully completed have the status ‘Completed’. The classifying factor between studies for supervised model training is their overall status being either in success class or in failure class. Terminated and withdrawn studies are labelled as “failure” and completed studies are labelled as “success”.

Numerical and categorical feature encoding. The final feature set is a mixture of numerical and categorical columns which requires different methods of encoding for a machine learning model. Large public datasets come with a lot of missing and erroneous data. Especially for numerical features, handling of the missing values could have a big impact on the performance of machine learning algorithms. Multiple Imputation by Chained Equations (MICE) algorithm selected to handle numerical missing data as it is a robust and informative method ¹¹. Missing cells imputed through an iterative sequence of predictive models where in each iteration one of the features imputed using other features of the dataset. This algorithm runs until it converges and all the numerical features values that are missing are imputed in this process.

One hot encoding is an effective method to encode categorical features. This method generates new binary features for each sub-category of a categorical feature The method handles missing categorical values by encoding them as zeros.

Train / test datasets. Phase specific datasets for Phase 1, Phase 2 and Phase 3 studies were generated for comparison to each other and the overall dataset. It is also possible to generate disease category specific datasets, but this is out of scope for this study. In order to estimate the performance of our machine learning models, the train-test split method was used ¹². For the final model a 70:30 train to test split ratio was selected. The train set is used to train the models whereas the test set is held aside for the final evaluation of the model. This is an effective and fast approach to test our trained models with data they have never seen before.

Handling data imbalance. The ratio of positive to negative class labelled studies changes for different subsets generated but data imbalance is a common issue for all subsets. The ratio of positive to negative samples for the overall dataset which contains studies from all phases is 15:85. Input data has bias towards the positive class, which can result in a falsely perceived positive effect on the model accuracy. In order to prevent this, random undersampling is applied to the training set. According to the given ratio, a necessary number of data points is deleted from the positive class subset. We use a 1 to 1 ratio between negative and positive class. Random under sampling was applied only on training samples after the train test split during the cross-validation process. The test set remained unsampled to preserve a realistic test distribution.

Top Feature Selection. Eligibility criteria features increased the feature set size significantly. In order to achieve the best performance without generating unnecessary noise in the data, feature selection was applied for the various datasets generated from the main dataset. An ablation study done to understand the effects of adding more features to the model performance. Figure 3 shows an example plot of number of features vs model error on test set using Phase 2 studies dataset. The purpose of plots is to find an elbow point. The elbow point is where the decrease angle of the error line dropped significantly so that we know adding more features does not have a significant effect on the performance. Analysing the plot in Figure 3, we can choose the elbow point as 4000 features. The same ablation study was conducted for all datasets in this paper and number of features selected accordingly.

Machine learning models. Logistic regression, random forest classifier and extreme gradient boosting classifier are selected for experiments. The logistic regression classifier is a simpler algorithm compared to the tree-based ensemble models such as random forest and extreme gradient boosting ^13,14. Even though feature selection is applied, the final datasets are still large sparse datasets. This ruled out many machine learning model architectures. Cross validation is used for the model evaluation to achieve unbiased metric scores. Using 5- old cross validation, the dataset split into 5 chunks and the model trained 5 times where each time a different chunk was kept as the test set to evaluate the model. This provided reliable metric scores to evaluate different models.

Tree based models require careful hyperparameter tuning, however it is computationally expensive to test out every combination of parameters to achieve the best results. Therefore, a strategy is made to find the best possible parameters for the models in hand. In order to prevent overfitting, the initial method is to control the model complexity and the second method is to add randomness to make training robust to noise ¹⁵. Optimal parameters are chosen after several iterations following this strategy.

Model interpretations using Shapley Additive exPlanations. SHAP (SHapley Additive exPlanations) is a framework based on Shapley values, a game theory approach ¹⁶. This method is used to get visual outputs to explain model predictions ¹⁷. SHAP locally explains the feature contributions on individual predictions by connecting optimal credit allocation to local explanations using Shapley values. A base value and an output value calculated for each plot. Base value is the average model output based on the training data and output value is the overall addition of the Shapley values for each feature for that instance. This allows us to explain the influence of features to the overall prediction.

Eligibility criteria search features results. McNemar’s Test was used as a statistical test used to compare predictive accuracy of two models ¹⁸. This test is used to compare two models, one trained with the original dataset features without the addition of 12,864 eligibility criteria search features, and the other with these features added on the original feature set. To apply this test xgBoost models were used and both models trained on the original dataset where studies from all phases included.

Our null hypothesis is that none of the two models performs better than the other. Conducting the McNemar test, we get a chi-squared value of 62.27 and p-value of 2.98e-15. The p value calculated is lower than the significance level 0.01, hence the null hypothesis is rejected. This illustrates that the addition of eligibility criteria search features is statistically significant with a level of confidence p<<0.01.

Machine learning classification results. Table 2 shows the comparison of predictive performances of three different machine learning classifiers. The evaluation metric scores calculated using 5-fold cross validation and xgBoost selected as the best performing classifiers.

Training on phase specific datasets results. Table 3 shows evaluation metric scores for xgBoost models trained on phase specific dataset and all phases dataset. Since the test data set aside is imbalanced, it is significant to evaluate these models using metrics that are independent of class distribution. Model 1 and Model 4 performed similarly and have the highest balanced accuracy scores. On the other hand, Model 2 has the highest F1 score for class 0. In this predictive model, one class is more important than the other because we are trying to predict trial termination which is labelled as Class 0. Hence, a higher score on the F1 score for class 0 indicates that Model 2 performs better taking into consideration the class imbalance. Model 3 performance is the poorest in this analysis of models.

Table 2 Evaluation metric scores for different classifiers; Logistic Regression, Random Forest Classifier, xgBoost Classifier trained in all phases dataset

Classifier/Score	ROC-AUC	Balanced Accuracy	F1 score	Class Accuracies (class 0 / class1)
Logistic Regression Classifier	71%	66%	36%	67% / 65%
Random Forest Classifier	76%	69%	40%	71% / 68%
xgBoost Classifier	78%	71%	43%	69% / 73%

Table 3 Evaluation metric scores for xGboost model trained on phase specific dataset

	ROC-AUC	Balanced Accuracy	F1 score - Class 0	Class Accuracies (class 0 / class1)
Model 1 - trained on Phase 1	78%	72%	42%	73% / 70%
Model 2 - trained on Phase 2	72%	66%	49%	63% / 69%
Model 3 - trained on Phase 3	72%	66%	40%	66% / 67%
Model 4 - trained on All phases	78%	70%	42%	70% / 70%

Comparison with state-of-the-art models. Table 4 demonstrates a comparison of the state-of-the-art models in this domain to ours. All the models used in this comparison are trained using data that includes studies from any phase. Therefore, the xgBoost model that trained on all phases dataset was used for this comparison. Similar to the comparison in the previous section, the most important aspect of model evaluation is the consideration of the data imbalance. First model on the table generated by Follett et al. performed poorly based on the gap between F1 score for failed class and the overall accuracy ³. Comparing the model generated by Elkin et al. and ours, the balanced accuracy and overall accuracies are quite similar, but our model shows much higher scores for ROC-AUC and F1 Score for class 0 ⁴. This indicates that our model is less biased considering the data imbalance in this dataset and performs better to predict early trial termination.

Table 4 Evaluation metric scores for state-of-the-art trial termination prediction models vs ours

	ROC-AUC	Balanced Accuracy	F1 Score	Accuracy
Follett et al. (study characteristics + mining Description field)	-	-	4%	93%
Elkin et al. (study characteristics + Description field Doc2Vec)	63.92%	67.20%	31.21%	73.01%
Our models (study characteristics + EC features)	78%	70%	42%	70% / 70%

Interpreting Test Instances with SHAP. xgBoost model trained on Phase 2 dataset selected for further investigation using SHAP library. Phase 2 dataset consists of 8880 data points; in order to have a balanced row to column ratio further feature selection is applied to select the best 1000 features. 3 study indices selected randomly from our test set that the trained model had never seen before to visualise the prediction interpretation using SHAP values.

In the SHAP plots for 3 different test instances, pink coloured features indicate a positive contribution whereas the blue colour indicates a negative contribution towards the calculated outcome. The overall value is calculated by adding up negative and positive contributions compared to the base value calculated for this model. The base value for this model is -0.00088. The base value is shown in grey text on the scale and the calculated value is shown in bold black on top of the scale for each plot. Figure 4 (a) and (b) show successful predictions for test cases that belong to Class 0 and Class 1, respectively. Figure 4 (c) shows an unsuccessful prediction where the actual class of the test item and the predicted class are not the same.

As Figure 4 (a) illustrates, the number of countries, inclusion criteria count value and eligibility criteria features 6236 (exclusion - Condition / inflammation) and 7753 (exclusion - Condition / dry eye) contributed positively whereas eligibility criteria feature 4810 (exclusion - Temporal / active) and non-acceptance of healthy volunteers contributed negatively. The overall value calculated is 0.23 which is larger than the base value of the trained model so the outcome prediction is Class 1, which is the accurate classification for this instance. Similarly Figure 4 (b) shows an accurate prediction for Class 0 where the calculated contributions add up to -0.71, hence the predicted class is Class 0 same as the actual class.

Figure 4 (c) illustrates an example where the model made a bad prediction. The overall calculated value is 0.01 so the class of the instance predicted as Class 1 whereas the true class was Class 0. The calculated value and the base value are quite similar but if we take a deeper look at this example, some of the eligibility criteria features that are among the largest contributions for the prediction are slightly illogical. 8726 (exclusion - Condition / addiction) contributed positively and 5200 (exclusion - Condition / inability) contributed negatively which both are intuitive explanations for the prediction whereas 582 (Inclusion - Mood / confirmed) and 46 (inclusion - Person / age) are not as intuitive as it is harder to understand the implications they have for the study protocol clearly. This illustrates that although for some examples SHAP interpretability is an intuitive approach to make suggestions on clinical trial design, there is still room for improvement especially on eligibility criteria feature engineering.

Limitations and Future Work. We used an unstructured eligibility criteria field to generate eligibility criteria search features with the help of the CHIA annotated criteria dataset for 1000 Phase 4 studies. It is possible that this dataset does not cover some of the main eligibility criteria categories and entities for earlier phase studies. Furthermore, some of the eligibility criteria search pairs generated from the CHIA dataset are not intuitive as the others. There is room for development regarding the generation of search pairs. Once these search features are extended for all phase studies and a more complex processing applied to get the most meaningful search features, it is expected that the predictive power of the trained models will increase, and more intuitive suggestions can be made for the study protocol by interpreting these models.

In this study, we extracted 420,268 clinical trials from the AACT database and used feature engineering methods on numerical, categorical and free-text columns and used machine learning to predict early trial termination using these features. We proposed adding eligibility criteria search features generated from free text columns in our dataset on top of the study characteristics features and showed that this approach is statistically significant to increase the predictive performance. We achieved 78% ROC-AUC, 70% balanced accuracy and 42% as the F1 score on the xgBoost model trained using all phases dataset. Finally, we used SHAP explanations to transform our black-box machine learning models into insightful suggestions towards the clinical study protocols. Eligibility criteria and design features are flagged as contributions towards success or failure during the interpretation process. These flagged features might help for preparing in advance for potential recruitment issues as well as suggesting direct changes to the study design if possible. Hence, this pipeline provides an optimised machine learning based trial design process.

Author Information
1. Lindus Health, London, UK

Ece Kavalci

[email protected]

2. Lindus Health, London, UK

Anthony Hartshorn

[email protected]

Author Contributions

E.K. worked on data collection and analysis. E.K and A.H worked on machine learning modelling and evaluation. E.K drafted the paper and A.H. revised the paper; all authors approved the final version of the manuscript.

Data Availability Statement

As mentioned in the Data Source section, clinicaltrials.gov is a public dataset and AACT is the public database version of clinicaltrials.gov. Similarly, CHIA dataset used in this project is a public dataset and referenced in the corresponding section of the paper.

Additional Information

Correspondence and requests for materials should be addressed to E.K. The author(s) declare no competing interests.

Fogel, D. B. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review. Contemp Clin Trials Commun 11, 156–164 (2018).
ClinicalTrials.gov Background - ClinicalTrials.gov. https://clinicaltrials.gov/ct2/about-site/background.
Follett, L., Geletta, S. & Laugerman, M. Quantifying risk associated with clinical trial termination: A text mining approach. Inf Process Manag 56, 516–525 (2019).
Elkin, M. E. & Zhu, X. Predictive modeling of clinical trial terminations using feature engineering and embedding learning. Scientific Reports 2021 11:1 11, 1–12 (2021).
AACT Database | Clinical Trials Transformation Initiative. https://aact.ctti-clinicaltrials.org/.
Chaturvedi, N. et al. Some data quality issues at ClinicalTrials.gov. Trials 20, 1–8 (2019).
Cummings, S. R. Clinical Trials Without Clinical Sites. JAMA Intern Med 181, 680–684 (2021).
Vetter, T. R. & Mascha, E. J. Defining the Primary Outcomes and Justifying Secondary Outcomes of a Study: Usually, the Fewer, the Better. Anesth Analg 125, 678–681 (2017).
Bodicoat, D. H. et al. Promoting inclusion in clinical trials—a rapid review of the literature and recommendations for action. Trials 22, (2021).
Kury, F. et al. Chia, a large annotated corpus of clinical trial eligibility criteria. Scientific Data 2020 7:1 7, 1–11 (2020).
van Buuren, S. & Groothuis-Oudshoorn, K. mice: Multivariate Imputation by Chained Equations in R. J Stat Softw 45, 1–67 (2011).
Dobbin, K. K. & Simon, R. M. Optimally splitting cases for training and testing high dimensional classifiers. BMC Med Genomics 4, 31 (2011).
Breiman, L. Random Forests. Machine Learning 2001 45:1 45, 5–32 (2001).
Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 13-17-August-2016, 785–794 (2016).
Notes on Parameter Tuning — xgboost 1.6.2 documentation. https://xgboost.readthedocs.io/en/stable/tutorials/param_tuning.html.
Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence 2020 2:1 2, 56–67 (2020).
Welcome to the SHAP documentation — SHAP latest documentation. https://shap.readthedocs.io/en/latest/index.html.
McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 1947 12:2 12, 153–157 (1947).

No competing interests reported.

SupplementaryMaterials.docx

Download PDF

Journal Publication

published 04 Jan, 2023

Read the published version in Scientific Reports →

Editorial decision: Major revision
24 Nov, 2022
Reviews received at journal
24 Nov, 2022
Reviews received at journal
16 Nov, 2022
Reviewers agreed at journal
06 Nov, 2022
Reviewers agreed at journal
07 Sep, 2022
Reviewers invited by journal
31 Aug, 2022
Editor assigned by journal
31 Aug, 2022
Editor invited by journal
31 Aug, 2022
Submission checks completed at journal
31 Aug, 2022
First submitted to journal
26 Aug, 2022

You are reading this latest preprint version

Improving Clinical Trial Design Using Interpretable Machine Learning Based Approach

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Methods

Results And Discussion

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1