This preprint is under consideration at BMC Bioinformatics. A preprint is a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Deep Active Learning for Classifying Cancer Pathology Reports
Shang Gao
Oak Ridge National Laboratory
Corresponding Author
ORCiD: https://orcid.org/0000-0003-1803-1457
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background: Automated text classification has many important applications in the clinical setting; however, obtaining labelled data for training machine learning and deep learning models is often difficult and expensive. Active learning techniques may mitigate this challenge by reducing the amount of labelled data required to effectively train a model. In this study, we analyze the effectiveness of eleven active learning algorithms on classifying subsite and histology from cancer pathology reports using a Convolutional Neural Network (CNN) as the text classification model.
Results: We compare the performance of each active learning strategy using two differently sized datasets and two different classification tasks. Our results show that on all tasks and dataset sizes, all active learning strategies except diversity-sampling strategies outperformed random sampling, i.e., no active learning. On our large dataset (15K initial labelled samples, adding 15K additional labelled samples each iteration of active learning), there was no clear winner between the different active learning strategies. On our small dataset (1K initial labelled samples, adding 1K additional labelled samples each iteration of active learning), marginal and ratio uncertainty sampling performed better than all other active learning techniques. We found that compared to random sampling, active learning strongly helps performance on rare classes by focusing on underrepresented classes.
Conclusions: Active learning can save annotation cost by helping human annotators efficiently and intelligently select which samples to label. Our results show that a dataset constructed using effective active learning techniques requires less than half the amount of labelled data to achieve the same performance as a dataset that constructed using random sampling.
Keywords
Active Learning, Cancer Pathology Reports, Information Extraction, Text Classification, Deep Learning, Convolutional Neural Networks
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Due to technical limitations, full-text HTML conversion of this manuscript could not be completed. However, the latest manuscript can be downloaded and accessed as a PDF.
This is a list of supplementary files associated with this preprint. Click to download.
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 1
Posted 24 Sep, 2020
No community comments so far
Editorial decision: Major revision
On 20 Oct, 2020
Review #2 received
Received 19 Oct, 2020
Review #1 received
Received 02 Oct, 2020
Reviewer #3 agreed
On 01 Oct, 2020
Reviewers invited
Invitations sent on 30 Sep, 2020
Reviewer #1 agreed
On 30 Sep, 2020
Reviewer #2 agreed
On 30 Sep, 2020
Editor assigned
On 07 Sep, 2020
Submission checks complete
On 06 Sep, 2020
Editor invited
On 06 Sep, 2020
First submitted
On 03 Sep, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Bioinformatics
Learn more about our company.
This preprint is under consideration at BMC Bioinformatics. A preprint is a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Deep Active Learning for Classifying Cancer Pathology Reports
Shang Gao
Oak Ridge National Laboratory
Corresponding Author
ORCiD: https://orcid.org/0000-0003-1803-1457
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Under Review
Version 1
Posted 24 Sep, 2020
No community comments so far
Editorial decision: Major revision
On 20 Oct, 2020
Review #2 received
Received 19 Oct, 2020
Review #1 received
Received 02 Oct, 2020
Reviewer #3 agreed
On 01 Oct, 2020
Reviewers invited
Invitations sent on 30 Sep, 2020
Reviewer #1 agreed
On 30 Sep, 2020
Reviewer #2 agreed
On 30 Sep, 2020
Editor assigned
On 07 Sep, 2020
Submission checks complete
On 06 Sep, 2020
Editor invited
On 06 Sep, 2020
First submitted
On 03 Sep, 2020
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Bioinformatics
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background: Automated text classification has many important applications in the clinical setting; however, obtaining labelled data for training machine learning and deep learning models is often difficult and expensive. Active learning techniques may mitigate this challenge by reducing the amount of labelled data required to effectively train a model. In this study, we analyze the effectiveness of eleven active learning algorithms on classifying subsite and histology from cancer pathology reports using a Convolutional Neural Network (CNN) as the text classification model.
Results: We compare the performance of each active learning strategy using two differently sized datasets and two different classification tasks. Our results show that on all tasks and dataset sizes, all active learning strategies except diversity-sampling strategies outperformed random sampling, i.e., no active learning. On our large dataset (15K initial labelled samples, adding 15K additional labelled samples each iteration of active learning), there was no clear winner between the different active learning strategies. On our small dataset (1K initial labelled samples, adding 1K additional labelled samples each iteration of active learning), marginal and ratio uncertainty sampling performed better than all other active learning techniques. We found that compared to random sampling, active learning strongly helps performance on rare classes by focusing on underrepresented classes.
Conclusions: Active learning can save annotation cost by helping human annotators efficiently and intelligently select which samples to label. Our results show that a dataset constructed using effective active learning techniques requires less than half the amount of labelled data to achieve the same performance as a dataset that constructed using random sampling.
Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Due to technical limitations, full-text HTML conversion of this manuscript could not be completed. However, the latest manuscript can be downloaded and accessed as a PDF.