Machine learning to predict ceftriaxone resistance using single nucleotide polymorphisms within a global database of Neisseria gonorrhoeae genomes

doi:10.21203/rs.3.rs-1999855/v1

Download PDF

Research Article

Machine learning to predict ceftriaxone resistance using single nucleotide polymorphisms within a global database of Neisseria gonorrhoeae genomes

https://doi.org/10.21203/rs.3.rs-1999855/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Antimicrobial resistance in Neisseria gonorrhoeae is an urgent global health issue ¹. Resistance to ceftriaxone, the mainstay of gonorrhea treatment, is increasing^2,3. Many genotypic mutations are associated with decreased susceptibility to ceftriaxone ^4,5. In this study, N. gonorrhoeae genomes from the PathogenWatch database were downloaded and used to train and test different machine learning (ML) models to predict ceftriaxone susceptibility/decreased susceptibility (S/DS). We evaluated seven different ML algorithms with 97 SNPs that are known to be associated with ceftriaxone resistance. After identifying the ML algorithm with the highest performance metrics, the impact score of individual SNPs were calculated. The algorithm was then retrained using various combinations of top scoring SNPs to measure performance. The study identified 5 SNPs that performed well to predict decreased susceptibility and might be promising targets for molecular assays to predict S/DS to ceftriaxone. The ML methods reported here might have applications for predicting AMR within other pathogens.

Neisseria gonorrhoeae, the bacterial pathogen that causes gonorrhea, is one of the most common sexually transmitted infections, with an estimated 82.4 million new infections globally in 2020.¹ Antimicrobial resistance (AMR) in N. gonorrhoeae is increasing and is considered an urgent global health issue.⁶ Currently, ceftriaxone is the last remaining empiric treatment option for gonorrhea, and ceftriaxone monotherapy is recommended in many settings, including the USA and UK.^2,3

Next-generation sequencing has revolutionized our understanding of bacterial pathogens, in general, and has advanced our knowledge of N. gonorrhoeae, in particular, including the determination of AMR mutations, investigating outbreaks, and surveillance.^7–9 The establishment of large databases of genomic sequence data provide a rich resource for public health professionals and researchers seeking to understand trends of N. gonorrhoeae on a global scale. One such database is PathogenWatch (https://pathogen.watch/), a publicly available database that combines sequence data with metadata, including phenotypic AMR data, that can be used not only for surveillance purposes, but also for the development of molecular assays to improve diagnosis and treatment.^10,11

Using bacterial genomic data, prior research has used machine learning (ML) algorithms to predict AMR in various pathogens. For example, Nguyen et al. used extreme gradient boosting to predict MIC values for Nontyphoidal Salmonella species against multiple antibiotics.¹² An analysis of different ML models for predicting AMR in different combinations of bacterial species and antibiotics, one of which included N. gonorrhoeae with ciprofloxacin and azithromycin, found performance varied depending on the species, resistance metrics, and antibiotic drugs, highlighting the complexity of developing clinically applicable ML models.¹³ When training an ML model, two types of input data can be used with genomic data, namely k-mer-based and reference-based.¹⁴ A k-mer method has an advantage when the clinical reference is not set and if pathogens have complex AMR mechanisms. On the other hand, the reference-based method incorporates well-established prior knowledge such as certain mutations in AMR genes.

N. gonorrhoeae is a pathogen that has been studied extensively, and many mutations are known to be associated with resistance to ceftriaxone.^4,5 Recently, Demczuk et al. used multivariate regression to create an equation for predicting minimum inhibitory concentration (MIC) values of N. gonorrhoeae using a number of antibiotics, including ceftriaxone, within a dataset of Canadian isolates.^12,15 However, machine learning approaches using genotypic data to predict AMR to ceftriaxone have not yet been accomplished. The objectives of this study were to use the global PathogenWatch database to develop, evaluate, and compare several different ML algorithms that use genotypic data to predict susceptibility/decreased-susceptibility of N. gonorrhoeae to ceftriaxone.

In total, there were 12,936 genome sequences extracted from PathogenWatch and 9,540 sequences with MIC data included in the ML analyses (Fig. 1a). Among those in the ML analyses, most N. gonorrhoeae sequences were from the USA and other high-income countries (Fig. 1b, Extended data Fig. 1). In total, 368 (0.04%) strains were associated with ceftriaxone MICs >0.064 mg/L, and classified as decreased-susceptible. Synthetic minority oversampling technique (SMOTE) was used to generate additional data for 8,804 DS synthetic sequences, resulting in 18,344 total sequences (9,171 susceptible + 368 decreased-susceptible + 8,804 decreased-susceptible^SMOTE). (Extended data Fig. 2)

We evaluated seven different ML algorithms on 18,344 real and synthetic ceftriaxone S/DS N. gonorrhoeae sequences. The AUC and bAcc scores for each model are depicted in Fig. 2a and b, while other score measures such as precision (0.723-0.903), recall (0.611-0.976), accuracy (0.755-0.926), and F-1 score (0.713-0.928) are listed in Table 1. The RFC model had the highest performance out of the seven ML models, having the top scores in precision (0.903), accuracy (0.926), F-1 score (0.928), AUC (0.966), and bAcc (0.926). Thus, the RFC model was selected for the prediction of ceftriaxone S/DS and further analysis.

To tease apart the RFC model, the SHAP values were calculated for each of the 97 SNPs initially used to train the model in order to measure the impact of each feature in the ML model (Fig. 2c). The RFC model was retrained with a different combination of these top SNPs. When the top 20 SNPs were used, a small decrease in AUC (0.966 to 0.941) and balanced accuracy (0.926 to 0.882) were observed compared to the model including all 97 SNPs. The models showed similar scores even when using only the top 5 SNPs: penA-501, penB-120, ponA-421, penB-121, and penA-545 (AUC: 0.915, and bACC: 0.875). (Fig. 4d-e; Table 2)

Using a large, global genomic database, we applied several ML models to predict decreased susceptibility to ceftriaxone in N. gonorrhoeae and found the RFC model performed best, with very high AUC and bACC values. Furthermore, analyzing SHAP values identified the SNPs most associated with ceftriaxone resistance and allowed for the identification of more efficient SNP combinations that performed comparably with the models using all 97 SNPs. Identifying ML models to predict decreased susceptibility of ceftriaxone in N. gonorrhoeae and the SNPs highly predictive of S/DS within a global database of N. gonorrhoeae genomes are important findings that advance our understanding of ceftriaxone resistance and can be used for enhanced surveillance. Furthermore, the ML model can aid in the development of molecular assays to predict ceftriaxone resistance in clinical specimens.

The RFC performed best for predicting ceftriaxone resistance compared to the other six models evaluated in this study. One possible reason the RFC performed better than the other models is its ability to handle missing data and maintain accuracy, which are important features given the presence of partial or missing AMR genetic data due to the incompleteness of genome sequences.¹⁶ Despite the incompleteness of genomic data, the high bAcc and accuracy indicate the robustness of the RFC model. Still, other studies have shown that the performance of different ML algorithms may vary depending on which sequence and drug are targeted for training.¹⁴ Thus, while we observed the RFC was the best ML model for ceftriaxone resistance, other models might work better when evaluating other antibiotics or with other datasets. Additional research and evaluation of other ML models using different datasets or with different pathogens will expand our knowledge of ML approaches to predict AMR.

In this study, SHAP analysis was used to determine the impact of each SNP on ceftriaxone resistance and identified several SNPs that have a high impact. We also observed that a combination of the top five SNPs showed only a slight decrease in performance, maintaining AUC > 90% and bACC > 85%. Interestingly, the top 20 SNPs in the SHAP analysis were primarily comprised of mutations within the penA, penB, and promoter region of mtrR genes. Many of those mutations have been incorporated into prior molecular assays and algorithms, including our own published work.^17–19 While our prior work used the mosaic penA for prediction of S/DS to ceftriaxone, the current analysis did not find a strong impact of the mosaic penA.^18,20 This finding might indicate mosaicism in penA is a less important factor for ceftriaxone decreased susceptibility on a global scale. Understanding how many SNPs, and in which combinations, can predict resistance to ceftriaxone is important, as they can be used develop molecular assays to predict ceftriaxone resistance. While antimicrobial susceptible testing (AST) is important for determining AMR, the process relies on bacterial culturing and has a long turnaround time. Therefore, incorporating genotypic markers into molecular assays can expedite the detection of AMR and lead to resistance-guided treatment.²¹ However, phenotypic AST remains critical to generating a full antibiotic susceptibility profile, for surveillance, and to advance our understanding of genotypic-phenotypic relationships, thus we do not envision molecular assays replacing AST entirely.

Although the ML models generated here show promising results, there are some limitations to this study. First, the distribution of isolates and AMR data are not equal, and a large proportion of data are from high-income countries. The model we trained may therefore be overfitted and somewhat region-specific. Second, even though SMOTE was incorporated to overcome the imbalance in S/DS strains in N. gonorrhoeae, there may be bias towards DS strains because, initially, there were only 368 genomes available. Third, we only used data included in the PathogenWatch database and were limited to the quality of data included in that database. However, the database is well-maintained and the data quality is understood to be high.¹⁰

In conclusion, our study generated a robust ML model to predict decreased susceptibility to ceftriaxone using global N. gonorrhoeae genomic data from PathogenWatch. We used a state-of-the-art ML technique to avoid overfitting the model and measured the relative impact of each mutation in known AMR genes. The results of this study go beyond simple identification of decreased susceptibility genetic mutations and can be used to guide the development of genotypic testing assays that could be incorporated into diagnostic tests or be used for the surveillance of AMR in N. gonorrhoeae. Moreover, the ML methods reported here could prove to be a foundational tool that can be applied to predicting AMR within other pathogens of interest. As AMR continues to increase, ML approaches to predict resistance can aid in the surveillance, diagnosis, and treatment of infections.

Data collection and preprocessing

For the machine learning analysis, a total of 12,936 N. gonorrhoeae genomes and relevant metadata were collected from PathogenWatch database on November 17, 2020. The 97 genetic mutations that are most associated with ceftriaxone resistance were extracted using an in-house python script called mutation detector (https://github.com/smha118/mutation_detector) as described previously.¹¹ The metadata for all strains were obtained from the PathogenWatch database including accession number, country information, and MIC values. Ceftriaxone susceptibility and decreased-susceptibility were determined based on MIC values of ≤ 0.064, and > 0.064 mg/L, respectively. Any strains without MIC values were excluded from further analysis (n = 3,396). Thus, a total of 9,540 strains were used for machine learning training and prediction. Before conducting the machine learning procedure, all of the nucleotides from the promoter region and amino acids were converted into quantifiable integers as defined in extended data Table 1. Similarly, ceftriaxone susceptibility and decreased-susceptibility were classified as 0 and 1 to represent susceptible and decreased-susceptible strains, respectively.

Machine learning (ML) training and prediction

Python (v3.8.12) was used as a primary coding language where Pandas (v1.2.4), NumPy (v1.20.3), Scikit-learn (v0.24.1), and Matplotlib (v3.4.3) were incorporated for data manipulation, matrices processing, ML analysis, and visualization, respectively.²² Since the data were largely imbalanced between susceptible and decreased-susceptible strains, synthetic minority oversampling technique (SMOTE), a method known to improve the accuracy of models trained compared to the oversampling with replacement method, was used to generate synthetic DS data using the k-nearest neighbor (KNN) method and oversample training/test data.²³ Following SMOTE, the dataset was split into training and test sets with 10-fold cross validation using the cross_validate function in scikit-learn. The 10-fold cross-validation technique allows for more accurate estimations of the area under the curve (AUC) and balanced accuracy (bAcc) scores while using all of the blocks of data as the test set.

A total of seven ML algorithms were utilized for evaluation, namely: logistic regression (LR), gaussian naive bayes classifier (GNB), k-nearest neighbors classifier (KNC), decision tree classifier (DTC), random forest classifier (RFC), gradient boosting classifier (GBC), and support vector machine (SVC). The performance of each ML model was evaluated based on precision\(\left(\frac{TP}{TP+FP}\right)\), recall (\(\frac{TP}{TP+FN}\)), accuracy ( \(\frac{TP+TN}{TP+FP+TN+FN}\)), balanced accuracy (bAcc; \(\frac{Specificty+Sensitivty}{2}\)), F-1 score (\(2\times \frac{precision \times recall}{precision + recall})\), and the area under the receiver operating characteristic curve (AUC).

Measuring of individual SNP scores in ML training

The individual SNP contributions for the RFC model were measured using SHapley Additive exPlanations (SHAP) values (v0.40.0).²⁴ SHAP values were calculated using the TreeExplainer and shap_values functions with the RFC model and training dataset as input data, respectively. Afterwards, the summary_plot function was used to visualize individual SNP contributions using the calculated SHAP values.

Role of the funding source

The funders of the study played no role in the study design; in the collection, analysis, or interpretation of data; in the writing of the report; or in the decision to submit the paper for publication.

Data availability

All the sequence data we used in this study is available at PathogenWatch database (https://pathogen.watch/).

Code availability

The code to generate and evaluate ML model is available on GitHub : https://github.com/smha118/ML_evalulation_tool.

Unemo, M. et al. WHO global antimicrobial resistance surveillance for Neisseria gonorrhoeae 2017–18: a retrospective observational study. The Lancet Microbe 2, e627–e636 (2021).
Alirol, E. et al. Multidrug-resistant gonorrhea: A research and development roadmap to discover new medicines. PLoS Med. 14, e1002366 (2017).
Fifer, H., Saunders, J., Soni, S., Sadiq, S. T. & FitzGerald, M. 2018 UK national guideline for the management of infection with Neisseria gonorrhoeae. Int. J. STD AIDS 31, 4–15 (2020).
Unemo, M. et al. Gonorrhoea. Nat. Rev. Dis. Prim. 5, 79 (2019).
Unemo, M. & Shafer, W. M. Antimicrobial resistance in Neisseria gonorrhoeae in the 21st century: past, evolution, and future. Clin. Microbiol. Rev. 27, 587–613 (2014).
Piszczek, J., St. Jean, R. & Khaliq, Y. Gonorrhea: Treatment update for an increasingly resistant organism. Can. Pharm. J. 148, 82–89 (2015).
Sánchez-Busó, L. et al. The impact of antimicrobials on gonococcal evolution. Nat. Microbiol. 4, 1941–1950 (2019).
Grad, Y. H., Goldstein, E., Lipsitch, M. & White, P. J. Improving Control of Antibiotic-Resistant Gonorrhea by Integrating Research Agendas Across Disciplines: Key Questions Arising From Mathematical Modeling. J. Infect. Dis. 213, 883–890 (2016).
Buckley, C. et al. Use of whole genome sequencing to investigate an increase in Neisseria gonorrhoeae infection among women in urban areas of Australia. Sci. Rep. 8, 1503 (2018).
Sánchez-Busó, L. et al. A community-driven resource for genomic epidemiology and antimicrobial resistance prediction of Neisseria gonorrhoeae at Pathogenwatch. Genome Med. 13, 61 (2021).
Adamson, P. C., Lin, E. Y., Ha, S.-M. & Klausner, J. D. Using a public database of Neisseria gonorrhoeae genomes to detect mutations associated with zoliflodacin resistance. J. Antimicrob. Chemother. 76, 2847–2849 (2021).
Nguyen, M. et al. Using Machine Learning To Predict Antimicrobial MICs and Associated Genomic Features for Nontyphoidal Salmonella. J. Clin. Microbiol. 57, (2019).
Hicks, A. L. et al. Evaluation of parameters affecting performance and reliability of machine learning-based antibiotic susceptibility testing from whole genome sequencing data. PLoS Comput. Biol. 15, e1007349 (2019).
Anahtar, M. N., Yang, J. H. & Kanjilal, S. Applications of Machine Learning to the Problem of Antimicrobial Resistance: an Emerging Model for Translational Research. J. Clin. Microbiol. 59, e0126020 (2021).
Demczuk, W. et al. Equations to predict antimicrobial MICs in Neisseria gonorrhoeae using molecular antimicrobial resistance determinants. Antimicrob. Agents Chemother. 64, (2020).
Petrazzini, B. O., Naya, H., Lopez-Bello, F., Vazquez, G. & Spangenberg, L. Evaluation of different approaches for missing data imputation on features associated to genomic data. BioData Min. 14, 44 (2021).
Lin, E. Y., Adamson, P. C., Deng, X. & Klausner, J. D. Establishing Novel Molecular Algorithms to Predict Decreased Susceptibility to Ceftriaxone in Neisseria gonorrhoeae Strains. J. Infect. Dis. 223, 1232–1240 (2021).
Lin, E. Y., Adamson, P. C., Ha, S.-M. & Klausner, J. D. Reliability of Genetic Alterations in Predicting Ceftriaxone Resistance in Neisseria gonorrhoeae Globally. Microbiol. Spectr. e0206521, (2022).
Peterson, S. W. et al. Molecular Assay for Detection of Genetic Markers Associated with Decreased Susceptibility to Cephalosporins in Neisseria gonorrhoeae. J. Clin. Microbiol. 53, 2042–2048 (2015).
Lin, E. Y., Adamson, P. C. & Klausner, J. D. Applying molecular algorithms to predict decreased susceptibility to ceftriaxone from a report of strains of Neisseria gonorrhoeae in Amsterdam, the Netherlands. The Journal of antimicrobial chemotherapy vol. 77 534–536 (2022).
Klausner, J. D. et al. Resistance-Guided Treatment of Gonorrhea: A Prospective Clinical Study. Clin. Infect. Dis. an Off. Publ. Infect. Dis. Soc. Am. 73, 298–303 (2021).
Pedregosa, F. et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011).
Chawla, N. V, Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic Minority over-Sampling Technique. J. Artif. Int. Res. 16, 321–357 (2002).
Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. CoRR abs/1705.0, (2017).

Acknowledgements

This study was supported by the National Institutes of Health (5T32GM008042-38 to EYL and R21AI157817 to JDK) and the Fogarty International Center (K01TW012170 to PCA). We would like to thank the UCLA Institute for Quantitative & Computational Biosciences (QCB) for their services.

Author information

Contributions

JK and PA conceived the study. SH and EL collected the data. SH developed the study protocol and analyzed data. EL verified the raw data and ceftriaxone resistance related SNPs. SH, PA, EL drafted the paper. All authors read and approved the final manuscript. All authors had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Competing interests

The authors declare the following competing interests: JDK has received consulting fees from Abbott, Cepheid, Roche, and Visby Medical in the past 12 months. All other authors declare no competing interests.

Table 1. The mean scores of ML models in 10-fold cross-validation.

Classifier	Precision	Sensitivity	Accuracy	AUC	bAcc*	F1 score
Logistic Regression	0.887317	0.916974	0.900467	0.945139	0.900526	0.901777
GaussianNaïve Bayesian	0.723291	0.975922	0.801246	0.886714	0.801896	0.830587
KNeighbors	0.884098	0.950897	0.913006	0.947449	0.913138	0.915965
DecisionTree	0.902933	0.953722	0.925779	0.962279	0.925884	0.927576
RandomForest	0.903394	0.954190	0.926246	0.965840	0.926352	0.928034
GradientBoosting	0.856833	0.611318	0.755452	0.883721	0.754918	0.713480
SVC	0.858429	0.936363	0.891121	0.936335	0.891289	0.895522

*bACC = Balanced Accuracy

Table 2. The mean scores of different combination of SNPs on RFC model in the 10-fold cross validation.

SNPs	Precision	Sensitivity	Accuracy	AUC	bAcc*	F1 score
All	0.903394	0.954190	0.926246	0.965840	0.926352	0.928034
Top20	0.851054	0.930304	0.884657	0.942909	0.885094	0.888592
Top10	0.840920	0.945451	0.882632	0.932152	0.882305	0.890096
Top5	0.867806	0.893909	0.878816	0.915591	0.878819	0.880605
Top3	0.772871	0.967863	0.841745	0.870983	0.841941	0.859361
Top1	0.807770	0.302236	0.613785	0.615572	0.614856	0.439649

*bACC = Balanced Accuracy

Yes there is potential Competing Interest. The authors declare the following competing interests: JDK has received consulting fees from Abbott, Cepheid, Roche, and Visby Medical in the past 12 months. All other authors declare no competing interests.

Extendeddata.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Machine learning to predict ceftriaxone resistance using single nucleotide polymorphisms within a global database of Neisseria gonorrhoeae genomes

Status:

Version 1

Abstract

Figures

Full Text

Discussion

Methods

Data collection and preprocessing

Machine learning (ML) training and prediction

Measuring of individual SNP scores in ML training

References

Declarations

Tables

Additional Declarations

Supplementary Files

Status:

Version 1