This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research Article
Sung Soo Ahn
Yonsei University College of Medicine
Corresponding Author
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Declarations:
Declarations
- This study involved human subjects.
- The author confirmed that all appropriate ethical guidelines for the use of human subjects have been followed, any necessary IRB and/or ethics committee review has been obtained, and information about the IRB/ethics committee is included in the manuscript.
- The author has confirmed that all necessary patient/participant consent or assent has been obtained and the appropriate institutional forms have been archived. If the IRB/ethics committee waived the requirement for patient/participant consent or assent, an explanation for the waiver is included in the text.
- The author has confirmed that a statement listing potential conflicts of interest or lack thereof is included in the text.
Objective: This study aims to determine how randomly splitting a dataset into training and test sets affects the estimated performance of a machine learning model under different conditions, using real-world brain tumor radiomics data.
Materials and Methods: We conducted two classification tasks of different difficulty levels with magnetic resonance imaging (MRI) radiomics features: (1) “Simple” task, glioblastomas [n=109] vs. brain metastasis [n=58] and (2) “difficult” task, low- [n=163] vs. high-grade [n=95] meningiomas. Additionally, two undersampled datasets were created by randomly sampling 50% from these datasets. We performed random training-test set splitting for each dataset repeatedly to create 1,000 different training and test set pairs. For each dataset pair, the least absolute shrinkage and selection operator model was trained by five-fold cross-validation (CV) or nested CV with or without repetitions in the training set and tested with the test set, using the area under the curve (AUC) as an evaluation metric.
Results: The AUCs in CV and testing varied widely based on data composition, especially with the undersampled datasets and the difficult task. The mean (±standard deviation) AUC difference between CV and testing was 0.029 (±0.022) for the simple task without undersampling and 0.108 (±0.079) for the difficult task with undersampling. In a training-test set pair, the AUC was high in CV but much lower in testing (0.840 and 0.650, respectively); in another dataset pair with the same task, however, the AUC was low in CV but much higher in testing (0.702 and 0.836, respectively). None of the CV methods helped overcome this issue.
Conclusions: Machine learning after a single random training-test set split may lead to unreliable results in radiomics studies, especially when the sample size is small.
Keywords
Artificial intelligence, Machine learning, Magnetic resonance imaging, Radiomics
Figure 1
Figure 2
Figure 3
Figure 4
This is a list of supplementary files associated with this preprint. Click to download.
- Supplementarymaterials.docx
Supplementary information for "Radiomics machine learning study with small sample size: single random training-test set split may result in unreliable results"
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
History
CURRENT STATUS: Posted
Version 2
Posted 04 Dec, 2020
No community comments so far
No comments provided
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Research Article
Sung Soo Ahn
Yonsei University College of Medicine
Corresponding Author
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
History
CURRENT STATUS: Posted
Version 2
Posted 04 Dec, 2020
No community comments so far
No comments provided
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Declarations:
Declarations
- This study involved human subjects.
- The author confirmed that all appropriate ethical guidelines for the use of human subjects have been followed, any necessary IRB and/or ethics committee review has been obtained, and information about the IRB/ethics committee is included in the manuscript.
- The author has confirmed that all necessary patient/participant consent or assent has been obtained and the appropriate institutional forms have been archived. If the IRB/ethics committee waived the requirement for patient/participant consent or assent, an explanation for the waiver is included in the text.
- The author has confirmed that a statement listing potential conflicts of interest or lack thereof is included in the text.
Objective: This study aims to determine how randomly splitting a dataset into training and test sets affects the estimated performance of a machine learning model under different conditions, using real-world brain tumor radiomics data.
Materials and Methods: We conducted two classification tasks of different difficulty levels with magnetic resonance imaging (MRI) radiomics features: (1) “Simple” task, glioblastomas [n=109] vs. brain metastasis [n=58] and (2) “difficult” task, low- [n=163] vs. high-grade [n=95] meningiomas. Additionally, two undersampled datasets were created by randomly sampling 50% from these datasets. We performed random training-test set splitting for each dataset repeatedly to create 1,000 different training and test set pairs. For each dataset pair, the least absolute shrinkage and selection operator model was trained by five-fold cross-validation (CV) or nested CV with or without repetitions in the training set and tested with the test set, using the area under the curve (AUC) as an evaluation metric.
Results: The AUCs in CV and testing varied widely based on data composition, especially with the undersampled datasets and the difficult task. The mean (±standard deviation) AUC difference between CV and testing was 0.029 (±0.022) for the simple task without undersampling and 0.108 (±0.079) for the difficult task with undersampling. In a training-test set pair, the AUC was high in CV but much lower in testing (0.840 and 0.650, respectively); in another dataset pair with the same task, however, the AUC was low in CV but much higher in testing (0.702 and 0.836, respectively). None of the CV methods helped overcome this issue.
Conclusions: Machine learning after a single random training-test set split may lead to unreliable results in radiomics studies, especially when the sample size is small.
Figure 1
Figure 2
Figure 3
Figure 4
This is a list of supplementary files associated with this preprint. Click to download.
- Supplementarymaterials.docx
Supplementary information for "Radiomics machine learning study with small sample size: single random training-test set split may result in unreliable results"
Loading...