Development of a Natural Language Processing Pipeline to Identify Histological Subtypes and Site of Cancer from Pathology Reports

doi:10.21203/rs.3.rs-2115286/v1

Download PDF

Research Article

Development of a Natural Language Processing Pipeline to Identify Histological Subtypes and Site of Cancer from Pathology Reports

https://doi.org/10.21203/rs.3.rs-2115286/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Purpose

To develop a Natural Language Processing (NLP) pipeline with the ability to determine the histological subtype and site of a patient’s cancer from pathology reports.

Methods

A Spark NLP-based deep learning model pipeline was developed to perform named entity recognition (NER) and assertion status detection for histological subtypes before extracting key relations of interest to determine the site of a patient’s cancer from pathology reports. We assessed the ability of this NLP pipeline to extract histological subtypes and site of a patient’s cancer against manual curation of pathology reports.

Results

A total of 1358 reports from 474 patients seen at a single tertiary cancer centre were used in the development and validation of the pipeline. The NLP pipeline achieved a mean accuracy of 99.79% and an F1 score of 84.08% for NER of histological subtypes. The relation extraction (RE) model also achieved an average accuracy of 91.96% and an F1-score of 92.45% for key entity relations relevant to histological subtypes entities.

Conclusion

We developed an NLP pipeline that can extract the histological subtypes and relate them to the site of a patient’s cancer from free-text pathology reports with high accuracy. This has the potential to be deployed for both research and clinical quality processes.

Spark

Natural language processing

Named entity recognition

Relation extraction

Medical texts

Biomedical NER

Natural language processing (NLP) is a range of computational techniques for the automatic analysis and representation of human language. Since its advent, its applications have extended from business to virtual assistant technologies(1). Its relevance is especially apparent in handling data in the medical field. It was estimated in 2017 that the footprint of medical data would double every 73 days by 2020 and continue to increase exponentially, and an estimated 80% of this data would be unstructured(2). However, this form of data remains largely inaccessible to statistical analysis(3, 4). Furthermore, manual extraction of this data is costly and time consuming(5, 6). One such example of unstructured data in healthcare are pathology reports. Such reports provide a wealth of knowledge that is useful in research and in prognostication(7). There have been models developed to extract such information using NLP, however, current models have had limited generalisability across oncologic subtypes and institutions(6, 8–13).

Spark NLP is a widely used library by healthcare organizations for NLP pipelines that are accurate and scale easily in a distributed environment(3). The Spark NLP Named Entity Recognition (NER) models use a BiLSTM-CNN-Char deep neural network architecture (14). This automatically detects word and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering steps(14). In this study, we aim to use machine learning to develop an efficient Spark NLP pipeline capable of extracting histological subtypes and site of origin of cancer from pathology reports. The identification of the histological subtypes and site of origin of cancer provides important foundational data elements in any cancer registry and is needed for use cases such as claims processing, clinical quality audits and clinical trial matching.

2.1. Data source and study participants

Deidentified pathology reports were retrieved (n = 1358) for 474 cancer patients seen at the National Cancer Centre Singapore between 21 January 1992 and 13 October 2021 who were also recruited into a prospectively maintained research databasePatients with Breast, Gastrointestinal, Gynaecologic and other types of tumours of all initial stages were included to reflect a range of histological subtypes to allow generalizability of the NLP pipeline to various solid tumours (Table 1a and 1b).

Table 1a

Primary tumour site of training and testing cohort. NER: Named entity recognition; RE: relation extraction.
Tumour Type		NER		RE		Cohort
Tumour Type		Training	Testing	Training	Testing	Cohort
Breast		95	25	90	21	97
Gastrointestinal		237	57	218	37	261
	Colorectal	87	34	83	17	72
	Pancreas	150	23	135	20	189
Gynaecologic		103	31	100	28	103
Other		1	0	1	0	10
Unknown		0	0	0	0	3
Total		436	113	409	86	474

Table 1b

Initial tumour stage of training and testing cohort. NER: Named entity recognition; RE: relation extraction.
Stage			Breast	Gastrointestinal	Gynaecologic	Other	Total
I	Cohort		14	16	25	0
	NER	Training	6	8	19	0
	NER	Testing	0	3	7	0
	RE	Training	6	8	17	0
	RE	Testing	0	1	6	0
II	Cohort		25	84	8	0
	NER	Training	12	63	6	0
	NER	Testing	4	16	2	0
	RE	Training	10	62	5	0
	RE	Testing	6	10	2	0
III	Cohort		19	75	26	3
	NER	Training	10	37	42	0
	NER	Testing	1	11	11	0
	RE	Training	9	30	42	0
	RE	Testing	2	4	7	0
IV	Cohort		34	84	43	10
	NER	Training	65	128	35	1
	NER	Testing	20	27	10	0
	RE	Training	63	117	35	1
	RE	Testing	13	22	13	0
Missing	Cohort		5	2	1	0
	NER	Training	2	1	1	0
	NER	Testing	0	0	1	0
	RE	Training	2	1	1	0
	RE	Testing	0	0	0	0
Total	Cohort		97	261	103	13	474
	NER	Training	95	237	103	1	436
	NER	Testing	25	57	31	0	113
	RE	Training	90	218	100	1	409
	RE	Testing	21	37	28	0	86

2.2. Report curation

All retrieved pathology reports were screened by a research data management team (HJT, FLL) under the supervision of a medical oncologist (RSYCT) and included date of pathology report, type of pathology report (i.e. Biopsy or surgical report), as well as the full pathology report. Pathology reports which did not contain at least the “Diagnosis” section which summarised the main findings of the pathology report were excluded. The selected reports were annotated by medical research staff using the INCEpTION open-source semantic annotation platform(15) before being reviewed a consultant medical oncologist (RSYCT) who served as curator. These annotations were done according to a consensus classification guidelines (Online Resource 1) developed by the curation team.

2.3. NLP Model development and evaluation

Figure 1 depicts the overall workflow of the proposed NLP pipeline, which contains three key models:

NER model for labelling histological subtypes along with other relevant entities such as “anatomical descriptor”, “body part”, “direction”, “primary site” and “malignant pathological finding” among others (Online Resource 1)
Assertion status detection model for identified entities
RE model for extracting relations between histological subtypes and site of cancer among others (Online Resource 1)

We first parsed each note into the various sections of the report. We excluded the “Clinical History” and “Gross description” sections, using only the “Diagnosis” and “Microscopic Description” sections as they were more clinically relevant to the NLP task. We randomly divided annotated pathology reports at the report level into training (90%; 10% of which were validation sets) and testing (10%) sets while preserving representation of all tumour types. We then used a custom-trained NER model(3) with Spark NLP library clinical word embeddings and fine-tuned it to label “anatomical descriptor”, “body part”, “direction”, “histologic subtype” and “primary site” in the pathology reports.

The identified entities were then passed into an assertion status detector model which is a separate pretrained Spark NLP assertion model (assertion_dl_large) that labels entities to give different probability weights to whether a body part has cancer and the histological subtypes of the cancer if present. Lastly, relations were extracted from the list of positive assertions to identify the histological subtypes described in each report and relate them to site of cancer using a TensorFlow-based custom-trained Spark NLP RE model. The six key relations identified were “histo_subtype-anatomical_descriptor”, “histo_subtype-body_part”, “anatomical descriptor-body part”, “direction-of”, “malignant pathology finding-anatomical_descriptor” and “malignant pathology finding -body_part”. All model performances were evaluated using accuracy and F1-score (F1) to the NLP model’s ability to correctly detect named entities, assertion labels and relations when compared to manual curation.

3.1. Cohort description

The cohort consisted of 474 patients and 1358 reports. The primary tumour site and initial stage of the training and testing cohort are shown in Table 1a and Table 1b respectively. Reports included in the testing cohort were mutually exclusive of the reports initially used to train the NLP pipeline. The mean number of pathology number reports curated per patient was 2.9

3.2. Model Performance

The model was evaluated using several metrics, namely accuracy, precision, recall and F1-score. The NER model scored a mean accuracy of 99.79% and F1-score of 84.08% for the identification of the histological subtypes token. The full performance of the NER model of the NLP pipeline is summarized in Table 2 (See Online Resource 2 for the full set of results).

Table 2

Accuracy, Precision, Recall and F1-score of the NER model of the NLP pipeline for “anatomical description”, “body part”, “direction”, “histological subtype”, “primary site” and “malignant pathological finding”. Support refers to number of occurrences of the entity.
Named entity	Accuracy	Precision	Recall	F1-score	Support
B-anatomical_descriptor	0.9908	0.5821	0.7348	0.6496	362
B-body_part	0.9895	0.7588	0.7530	0.7559	656
B-direction	0.9973	0.9149	0.8113	0.8600	265
B-histologic_subtype	0.9980	0.8654	0.8133	0.8385	166
B-primary_site	0.9996	0.4474	0.6800	0.5397	25
I-anatomical_descriptor	0.9938	0.4926	0.6552	0.5624	203
I-body_part	0.9943	0.8255	0.7550	0.7887	351
I-direction	0.9993	0.8333	0.1562	0.2632	32
I-histologic_subtype	0.9979	0.8896	0.8012	0.8431	171
I-primary_site	0.9999	0.4545	0.4545	0.4545	11
B-malignant_pathological_finding	0.9906	0.6274	0.6710	0.6484	389
I-malignant_pathological_finding	0.9941	0.6172	0.6695	0.6423	236

The performance of the RE model of the NLP pipeline is summarised in Table 3. The RE module scored an overall accuracy of 91.96% and F1-score of 92.45% for the extraction of the six key relations.

Table 3

Accuracy, Precision, Recall and F1-score of the RE model of the NLP pipeline for six key relations relevant to histological subtype. Support refers to number of occurrences of the entity.
Relation	Accuracy	Precision	Recall	F1-score	Support
anat_descriptor-body_part	0.9944	0.9178	0.9178	0.9178	146
direction-of	0.9551	1.0000	0.9154	0.9558	130
histo_subtype-anat_descriptor	0.9794	0.9048	1.0000	0.9500	19
histo_subtype-body_part	0.9963	0.9420	0.8553	0.8966	76
malignant_path_finding-anat_descriptor	0.9720	0.9863	0.9231	0.9536	78
malignant_path_finding-body_part	0.9869	0.7981	0.9651	0.8737	86

Figure 2 depicts an example of the predictions made by this NLP pipeline, illustrating the interface that can be used to facilitate users to check the pipeline’s output for both NER tokens as well as relations.

The NLP pipeline developed in this study produced state-of-the-art results, scoring highly in accuracy and F1- score. To the best of our knowledge, this is the first time a NER – assertion – RE extraction pipeline has been applied to extract histological subtypes. Our model had comparable results to other NLP models that handled pathology reports (11, 13, 16, 17). One such model suggested by Abedian (2021) was created to extract staging and diagnosis information from surgical pathology reports such as ICD-10 codes and TNM staging regardless of the disease area. Compared to this model, our model had largely comparable results in terms of F1-score. Another model proposed by Yala (2017) used machine learning to parse breast pathology reports. Our model scored better in terms of overall accuracy.

This model has the advantage of being a single model that is applicable to multiple cancer types as compared to other models which were trained in a single cancer type(11, 17). Given that the model is Spark NLP based, the hardware compute requirements and run-times of the model are also significantly less compared to transformer based model architectures. This is important during deployment in healthcare settings where compute resources may be finite. Limitations of this study include limited generalizability to other institutions and difficulty performing when encountering variances in spelling and semantics. The applicability of this pipeline to other institutions remains to be seen as the reports used to train and validate this model were sourced from a single institution. While future validation studies in other institutions are required, our team believes it should perform well given the histological subtypes identified by the model generally follow standardized pathology terminology per WHO classification of tumours. This NLP pipeline could potentially have difficulty performing when encountering misspelt words, poor formatting (e.g. absence of spaces between words) and uncommon phrasing. These caveats should be noted by users and can be avoided or reduced if certain quality assurance processes such as spell and formatting checks are taken prior to release of pathology reports and appropriate pre-processing such as removing of excess white-space is done prior to running the NLP pipeline. The robustness of this model can be improved upon by using a larger and more diverse dataset. To this end, our team looks forward to participating in future multi-institutional collaborative efforts for NLP in healthcare.

We developed an NLP pipeline that can accurately extract histological subtypes and relate them to the site of a patient’s cancer from free-text pathology reports with high accuracy. This has the potential to be deployed in healthcare settings for both research and clinical quality processes.

6.1 Ethics approval

This study was approved by the Singapore Health Services’ Institutional Review Board (CIRB Ref: 2019/2401).

6.2 Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

6.3 Author’s Contributions

The conceptualization of this project was done by R.S.Y.C.T. and G.H.L.. Funding acquisition was done by I.T.B.H.. Methodology was done by R.S.Y.C.T. and G.H.L.. Data curation was done by R.S.Y.C.T. and S.B.T.. Formal analysis and investigation was performed by R.S.Y.C.T. and G.H.L.. R.S.Y.C.T. and F.S.E.N. wrote the original draft. All authors reviewed and edited the manuscript.

6.4 Funding

This research was supported by A*STAR under its IAF-PP programme (Award H18/01/a0/019).

6.5 Availability of data and materials

The data that support the findings of this study are available from the corresponding author, R.S.Y.C.T, upon reasonable request.

Cambria E, White B. Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]. IEEE Comput Intell Mag. 2014 May;9(2):48–57.
Ahmed MN, Toor AS, O’Neil K, Friedland D. Cognitive Computing and the Future of Health Care Cognitive Computing and the Future of Healthcare: The Cognitive Power of IBM Watson Has the Potential to Transform Global Personalized Medicine. IEEE Pulse. 2017 May;8(3):4–9.
Kocaman V, Talby D. Accurate Clinical and Biomedical Named Entity Recognition at Scale. Softw Impacts. 2022 Aug;13:100373.
Griffon N, Charlet J, Darmoni SJ, Section Editors for the IMIA Yearbook Section on Knowledge Representation and Management. Managing Free Text for Secondary Use of Health Data: Findings from the Yearbook 2014 Section on Knowledge Representation and Management. Yearb Med Inform. 2014 Aug;23(01):167–9.
Deleger L, Molnar K, Savova G, Xia F, Lingren T, Li Q, et al. Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. J Am Med Inform Assoc JAMIA. 2013;20(1):84–94.
Warner JL, Levy MA, Neuss MN, Warner JL, Levy MA, Neuss MN. ReCAP: Feasibility and Accuracy of Extracting Cancer Stage Information From Narrative Electronic Health Record Data. J Oncol Pract. 2016 Feb;12(2):157–8; e169-167.
McNiff K. The Quality Oncology Practice Initiative. J Oncol Pract. 2006 Jan;2(1):26–30.
McCowan I, Moore D, Fry MJ. Classification of cancer stage from free-text histology reports. Conf Proc Annu Int Conf IEEE Eng Med Biol Soc IEEE Eng Med Biol Soc Annu Conf. 2006;2006:5153–6.
Kim BJ, Merchant M, Zheng C, Thomas AA, Contreras R, Jacobsen SJ, et al. A natural language processing program effectively extracts key pathologic findings from radical prostatectomy reports. J Endourol. 2014 Dec;28(12):1474–8.
Leyh-Bannurah SR, Tian Z, Karakiewicz PI, Wolffgang U, Sauter G, Fisch M, et al. Deep Learning for Natural Language Processing in Urology: State-of-the-Art Automated Extraction of Detailed Pathologic Prostate Cancer Data From Narratively Written Electronic Health Records. JCO Clin Cancer Inform. 2018 Dec;2:1–9.
Yala A, Barzilay R, Salama L, Griffin M, Sollender G, Bardia A, et al. Using machine learning to parse breast pathology reports. Breast Cancer Res Treat. 2017 Jan;161(2):203–11.
Nguyen AN, Lawley MJ, Hansen DP, Bowman RV, Clarke BE, Duhig EE, et al. Symbolic rule-based classification of lung cancer stages from free-text pathology reports. J Am Med Inform Assoc JAMIA. 2010;17(4):440–5.
Abedian S, Sholle ET, Adekkanattu PM, Cusick MM, Weiner SE, Shoag JE, et al. Automated Extraction of Tumor Staging and Diagnosis Information From Surgical Pathology Reports. JCO Clin Cancer Inform. 2021 Dec;(5):1054–61.
Chiu JPC, Nichols E. Named Entity Recognition with Bidirectional LSTM-CNNs. Trans Assoc Comput Linguist. 2016 Jul 1;4:357–70.
Klie JC, Bugert M, Boullosa B, Eckart de Castilho R, Gurevych I. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In: Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations [Internet]. Santa Fe, New Mexico: Association for Computational Linguistics; 2018 [cited 2022 Aug 25]. p. 5–9. Available from: https://aclanthology.org/C18-2002
AAlAbdulsalam AK, Garvin JH, Redd A, Carter ME, Sweeny C, Meystre SM. Automated Extraction and Classification of Cancer Stage Mentions fromUnstructured Text Fields in a Central Cancer Registry. AMIA Jt Summits Transl Sci Proc AMIA Jt Summits Transl Sci. 2018;2017:16–25.
Glaser AP, Jordan BJ, Cohen J, Desai A, Silberman P, Meeks JJ. Automated Extraction of Grade, Stage, and Quality Information From Transurethral Resection of Bladder Tumor Pathology Reports Using Natural Language Processing. JCO Clin Cancer Inform. 2018 Dec 1;2:1–8.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Development of a Natural Language Processing Pipeline to Identify Histological Subtypes and Site of Cancer from Pathology Reports

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusion

Figures

1. Introduction

2. Methods

2.1. Data source and study participants

2.2. Report curation

2.3. NLP Model development and evaluation

3. Results

3.1. Cohort description

3.2. Model Performance

4. Discussion

5. Conclusion

6. Declarations

7. References

Additional Declarations

Supplementary Files

Status:

Version 1