Depression has surged in Korea, with 933,481 patients in 2021, a 35.1% increase since 2017. Globally, 5% of adults suffer from depression, resulting in over 700,000 suicides annually. However, Korea has only 29.5 mental health workers per 100,000 people, below the OECD average of 97.1. There is an increasing demand for mental illness diagnosis support systems internationally, and various research and development efforts are being attempted to alleviate mental illness. The problem of insufficient mental health human resources and treatment overload can be alleviated by medical artificial intelligence technology. We developed an artificial intelligence model for a clinical decision support system that determines the severity of depression using natural language data about depressive symptoms reported by patients treated in a psychiatric unit contained in our Clinical Data Warehouse (CDW). This study selected psychiatric depression patients from the Bundang CHA University Hospital CDW in South Korea between 2018 and 2022. Among them, 169 patients were diagnosed with mild depressive episodes, and 460 patients were diagnosed with moderate depressive episodes based on psychiatric symptom presentations. The control group utilized natural language datasets provided for artificial intelligence development on the AI Hub platform. The final analysis dataset consisted of Class 2: Moderate depression episode (460 patients), Class 1: Mild depression episode (169 patients), and Class 0: Normal (123,690 conversation sessions). Using this depression natural language dataset, we developed a model to classify depression severity. We applied various algorithms to accurately diagnose the severity of depression based solely on the symptoms reported by patients through psychiatric clinical texts, and selected the one with the highest numerical diagnostic accuracy and the best practical diagnostic classification. As a result, XGBoost showed the highest diagnostic accuracy, with an accuracy of 99.7%, precision of 99.6%, recall of 99.7%, and an F1 score of 99.6%. Additionally, the AUC was close to 1. Utilizing advanced medical artificial intelligence and natural language processing technology in the field of psychiatry can be greatly beneficial in assisting with the precise, personalized assessment of depression severity based on the content of what patients express.
Research Article
Development of a depression severity diagnosis model through natural language processing by psychiatry clinical texts
https://doi.org/10.21203/rs.3.rs-4805394/v1
This work is licensed under a CC BY 4.0 License
You are reading this latest preprint version
Natural language processing
Depression
eXplainable AI
Diagnosis model
Machine learning
Over the past five years, depression has been increasing rapidly in Korea, regardless of gender and age. The number of depression patients in Republic of Korea in 2021 is 933,481, an increase of 35.1% compared to 2017 [1]. Globally, an estimated 5% of adults suffer from depression (Approximately 280 million people in the world have depression) and more than 700,000 people die due to suicide every year [2], and this figure continues to increase. Meanwhile, as of 2017, the number of domestic mental health workers was 29.5 per 100,000 people, less than one-third of the average of 97.1 in OECD countries [3]. In this situation, artificial intelligence technology using large language models [4] such as ChatGPT [5] is developing rapidly. It has become possible to develop a clinical diagnosis support system using artificial intelligence technology globally. This has led to an era in which personalized precision medicine can be realized. There is an increasing demand for mental illness diagnosis support systems internationally, and various research and development efforts are being attempted to alleviate mental illness. Digital Therapeutics (DTx) [6] and Clinical Decision Support Systems (CDSS) [7] that include medical artificial intelligence, which have been released recently, can be said to be the result. The problem of insufficient mental health human resources and treatment overload can be alleviated by medical artificial intelligence technology. We developed an artificial intelligence model for a clinical decision support system that determines the severity of depression using natural language data about depressive symptoms reported by patients treated in a psychiatric unit contained in our Clinical Data Warehouse (CDW). This artificial intelligence technology can help medical staff focus on more in-depth treatment of depression than before by simplifying the diagnosis of depression and assisting in diagnosing the severity of depression quickly and accurately.
Data collection process
The subject selected 4,002 psychiatric depression patients from bundang CHA University hospital CDW in South Korea between 2018 and 2022. Among them, 169 patients were diagnosis mild depressive episodes, and 460 patients were diagnosis moderate depressive episodes based on psychiatric symptom presentations. The remaining 3,373 patients were excluded due to inadequate data or uncertainty in the diagnosis of depression severity. The control group was extracted from open-source data available on the AI Hub platform [8] in the Republic of Korea. The AI Hub platform is an open-source data platform provided in South Korea, offering various datasets for artificial intelligence and machine learning research. These datasets are primarily intended for tasks related to natural language processing, image analysis, speech processing, and other AI-related tasks. Researchers and developers can utilize publicly available data through the AI Hub platform to conduct various projects. The selected groups were combined to complete the dataset, consisting of 124,319 cases categorized into Class 2: Moderate depression episode (460 patients), Class 1: Mild depression episode (169 patients), and Class 0: Normal (123,690 conversation session) (Fig. 1. a).
Development process
Based on this data, we use text data on depression symptoms collected from a mental health clinic. Divide training data and test data in a ratio of 8:2. Quantify text data using TfidfVectorizer [9] from the sklearn library [10]. Based on numerical data, several machine learning algorithms are applied to learn the model. The algorithms used are Logistic Regression [11], Support Vector Machine [12], Random Forest [13], Multinomial Naïve Bayes [14], K-Nearest Neighbors [15], AdaBoost [16], XGBoost [17], and CatBoost [18]. Select the most suitable model among various models and save the model from pickle library [19]. Load the saved optimal model. Complaint text data about depression symptoms from a new mental health clinic is retrieved and applied to a model to diagnose the severity or normality of depression. Lastly, we use LIME (Locally Interpretable Model-agnostic Explanations) [20], an explainable AI (XAI), to provide causal explanations for the results of the model's predictions (Fig. 1. b).
Analysis & Development tools
All analyses & developments were performed using Python 3.10.9 [21] with Jupyter notebook [22].
Depression patients characteristics
Looking at the gender and age characteristics of depression patients extracted from the CDW, among the 460 patients with moderate depression, 172 were male, 23 were children under 12 (13.4%), and 44 were adolescents aged 13 to 19 (28.6%), 20–39. There were 52 people (30.2%) in the young age group, 40 people (23.3%) in the middle-aged age group of 40–64 years, 13 people (7.6%) in the elderly age group 65 years or older, and among 288 women, 16 people (5.6%) were children under 12 years of age. ), 60 people (20.8%) were young people aged 13–19, 76 people (26.4%) were young people aged 20–39, 104 people were middle-aged (36.1%) were 40–64 years old, and 32 people were aged 65 or older (11.1%). %). Among 169 people with mild depression, 72 were men, 0 (0.0%) were children under the age of 12, 9 (12.5%) were adolescents aged 13–19, and 27 (37.5%) were young adults aged 20–39, 40–64. There were 27 people (37.5%) in the middle-aged group, 9 people (12.5%) in the elderly group over 65 years old, 2 people (2.1%) in the 97 female group, children under 12 years old, and 7 people (7.2%) in the 13–19 year old group. ), 22 people (22.7%) were young people aged 20–39, 43 people (44.3%) were middle-aged people aged 40–64, and 23 people (23.7%) were aged 65 or older (Table 1).
Table 1. Depression patients characteristics
* Bundang CHA University Hospital Clinical Data Warehouse : Moderate & Mild Depression episode patients psychiatry clinical texts (2018-2022 years) – Patients Characteristics (Sex & Age)
Performance of depression severity diagnosis model
We applied various algorithms to make a precise diagnosis of the severity of depression based solely on the symptoms complained of by patients through psychiatric clinical texts, and selected the one that had the highest numerical diagnostic accuracy and the best practical diagnostic classification. As a result, XGBoost had the highest diagnostic accuracy with accuracy of 99.7%, precision of 99.6%, recall of 99.7%, and F1 score of 99.6% (Table 2). Also, AUC was close to 1 (Fig. 2).
XGBoost showed good practical diagnostic classification when the sentences were input. In the case of Logistic Regression and Support Vector Machine, the diagnostic accuracy was at the same level, but it was found that the diagnostic classification did not work well when sentences were actually entered.
|
Classifier |
Accuracy |
Precision |
Recall |
F1 Score |
|---|---|---|---|---|
|
XGBoost |
99.7 |
99.6 |
99.7 |
99.6 |
|
Logistic Regression |
99.7 |
99.6 |
99.7 |
99.6 |
|
Support Vector Machine |
99.7 |
99.6 |
99.7 |
99.6 |
|
CatBoost |
99.6 |
99.5 |
99.6 |
99.5 |
|
Multinominal Naïve Bayes |
99.6 |
99.1 |
99.6 |
99.3 |
|
Random Forest |
99.6 |
99.5 |
99.6 |
99.5 |
|
AdaBoost |
99.6 |
99.4 |
99.6 |
99.5 |
|
K-Nearest Neighbors |
63.7 |
99.5 |
63.7 |
77.4 |
Depression severity diagnosis
We finally adopted the XGBoost algorithm and attempted to diagnose depression severity. And as a result of checking the actual operating state of the model, it was confirmed that moderate depression and mild depression were well classified when a sentence indicating a depressed mental state was entered as follows.
|
Moderate Depression case 1 (Fig. 3. a) |
|---|
|
“As a housewife, I care a lot about my children. I stay mostly for my children, and I have a lot of worries about my son, who was discharged from the military two months ago. I am introverted and sensitive because I have a lot of worries about small things. In the past, I worried a lot about my mother's lung surgery and my father's decision to time the deadline.” |
|
• Moderate Prediction probabilities = 54% • Mild Prediction probabilities = 23% • Normal Prediction probabilities = 23% |
|
Moderate Depression case 2 (Fig. 3. a) |
|---|
|
“My heart was pounding even before that, but it got worse and worse a few days ago and whenever I'm alone, my anxiety hits me, so I can't hold on. I hate being alone and in the dark.” |
|
• Moderate Depression Prediction probabilities = 70% • Mild Prediction probabilities = 5% • Normal Prediction probabilities = 25% |
|
Mild Depression case 1 (Fig. 3. c) |
|---|
|
“Since I left my job, I have a lot of financial worries. I feel anxious every time I lie down to sleep. I can't control my thoughts. My husband is so incompetent that he has nothing for his age. If I get sick, I wonder if it will hurt my children. If I'm healthy, I can work next year when my unemployment benefits are over. I'm anxious about the future.” |
|
• Moderate Depression Prediction probabilities = 1% • Mild Depression Prediction probabilities = 88% • Normal Prediction probabilities = 10% |
|
Mild Depression case 2 (Fig. 3. c) |
|---|
|
“No matter how hard I work like a slave, my life doesn't improve, so it's very difficult to live because the future is dark, and I feel empty. My situation even makes me feel ashamed.” |
|
• Moderate Depression Prediction probabilities = 17% • Mild Depression Prediction probabilities = 45% • Normal Prediction probabilities = 37% |
In addition, it was confirmed that even when a sentence with a normal mental state without depression was entered, it was well classified as normal.
|
Normal case 1 (Fig. 3. e) |
|---|
|
“I'm trying to do hobbies to make my life a plus. Cooking, crosshairs, coffee roasting and productive hobbies seem to have more joy than just hobbies in many ways.” |
|
• Moderate Depression Prediction probabilities = 16% • Mild Depression Prediction probabilities = 2% • Normal Prediction probabilities = 91% |
|
Normal case 2 (Fig. 3. e) |
|---|
|
“You have to be able to stand up on your own to get happiness. You have to know how to govern yourself the best.” |
|
• Moderate Depression Prediction probabilities = 16% • Mild Depression Prediction probabilities = 6% • Normal Prediction probabilities = 79% |
In the psychiatry department, patients' psychological records in natural language are used for actual treatment. This suggests that mental illness can be diagnosed through natural language processing. Using natural language processing artificial intelligence, we developed an artificial intelligence model that diagnoses the severity of depression by learning a patient's symptom expression and corresponding diagnostic label from psychiatric clinical records.
This suggests that mental illness can be diagnosed through natural language processing. In psychiatry, the DSM-5 [23] criteria are generally used to diagnose depressive episodes (F32) [24] for patient treatment. Despite these standards, mental illness remains an area that cannot be diagnosed with clear clinical values like high blood pressure or diabetes, making diagnoses inherently subjective. Tools such as the nine-item Patient Health Questionnaire (PHQ-9) [25], Beck Depression Inventory (BDI) [26], and Hamilton Rating Scale for Depression [27] are used to assess the severity of depression. However, even when psychiatrists use accumulated experience and DSM-5 criteria, as well as these standardized scales, subjectivity cannot be entirely avoided. Additionally, these scale-based methods cannot fully capture the diverse manifestations of depressive symptoms in each individual.
This study has limitations in the following three areas. 1. For the precision of diagnosing depression severity in this study, we only extracted cases where the diagnosis at the follow-up visit matched the initial diagnosis. Therefore, the data size was inevitably reduced as we trained the AI model using the content of the patients' expressions of depression at their initial visits. 2. Unlike English, which is an isolating language where most morphemes function as individual words, Korean exhibited a phenomenon where performing morpheme analysis at the word level actually reduced the precision of diagnostic classification. As an agglutinative language, Korean forms words by attaching affixes to stems, which determine meaning and grammatical function. Consequently, it was found that performing morpheme analysis at the sentence level or vectorizing entire sentences for training resulted in more accurate diagnoses. Moreover, due to the nature of mental illness, where patients use different expressions compared to healthy individuals and exhibit logical incoherence, this phenomenon was further highlighted. In natural language processing for medical AI, it is necessary to consider different strategies for morpheme analysis based on language types, such as agglutinative, isolating, polysynthetic, and inflectional languages. 3. Patients with mental illnesses exhibit differences in various aspects of conversation, such as the progression of dialogue and vocabulary used, compared to the everyday expressions and content of healthy individuals. This poses challenges for the application and utilization of Large Language Models (LLMs). Consequently, this suggests the need for the development of specialized smaller Large Language Models (sLLMs) tailored for medical AI in psychiatry.
In conclusion, utilizing advanced medical artificial intelligence and natural language processing technology in the field of psychiatry can be greatly beneficial in assisting with the precise, personalized assessment of depression severity based on the content of what patients express.
Author Contributions Conceptualization: HW Han, Methodology: MG Kim, Analysis: MG Kim, CY Song, DW Wang, Data visualization: MG Kim, CY Song, Writing-original draft preparation: MG Kim, Writing-review and editing: HW Han
Funding This study received support from the following national research and development projects.
(Grant number: 20023719) Workers AI Assistant and Health Care Service for Asynchronous Work Crowdwalkers in Hybrid Work Environments funded By the Ministry of Trade, Industry & Energy (MOTIE, Korea). And this research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ICAN(ICT Challenge and Advanced Network of HRD) support program(RS-2024-00437880) supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation)
Data AvailabilityThe data that supports the findings are available upon request from the corresponding author.
Ethics statement We received review and approval from the Bundang Cha Hospital Institutional Review Board (CHAMC 2023-12-018) for the CDW data-based retrospective study and medical artificial intelligence development targeting psychiatric patients.
Conflict of Interest Statements None of the authors has any conflicts of interest to declare in relation to the work in this article.
Additional information Correspondence and requests for materials should be addressed to HyunWook Han
- Republic of Korea, Health Insurance Review & Assessment Service, Analysis of treatment status for depression and anxiety disorders over the past 5 years (2017-2021), (2022).https://www.hira.or.kr/bbsDummy.do?pgmid=HIRAA020041000100&brdScnBltNo=4&brdBltNo=10627&pageIndex=1
- World Health Organization, Fact sheets: Depressive disorder (depression), (2023).https://www.who.int/news-room/fact-sheets/detail/depression
- World Health Organization, Global Health Observatory, Human resources for mental health, (2017).
https://www.who.int/data/gho/data/themes/topics/indicator-groups/indicator-group-details/GHO/human-resources - OpenAI, Better Language Models and Their Implications, (2019)..
- OpenAI, ChatGPT, (2022). https://openai.com/chatgpt
- Joseph C Kvedar, Alexander L Fogel, Eric Elenko, Daphne Zohar Digital medicine's march on chronic disease. Nature Biotechnology, 34, 239–246, (2016). https://doi.org/10.1038/nbt.3495
- Reed T. Sutton, David Pincock, Daniel C. Baumgart, Daniel C. Sadowski, Richard N. Fedorak, Karen I. Kroeker, An overview of clinical decision support systems: benefits, risks, and strategies for success, Nature Digital Medicine, 3, 17, (2020). https://doi.org/10.1038/s41746-020-0221-y
- Republic of Korea, Ministry of Sciecne and ICT, National Information Society Agency, AI Hub, (2022).
https://www.aihub.or.kr/ - Olivier Grisel, Mathieu Blondel, Lars Buitinck, Robert Layton, Jochen Wersdörfer, Roman Sinayev, scikit-learn, TfidfVectorizer: `sklearn.feature_extraction.text` submodule gathers utilities to build feature vectors from text documents. (2024).
https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html - Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, Édouard Duchesnay, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12(85):2825−2830, (2011). https://jmlr.csail.mit.edu/papers/volume12/pedregosa11a/pedregosa11a.pdf
- Conklin, J. D., Applied Logistic Regression. Technometrics, 44(1), 81–82, (2022). https://doi.org/10.1198/tech.2002.s650
- Derek A. Pisner, David M. Schnyer, Chapter 6 - Support vector machine, Editor(s): Andrea Mechelli, Sandra Vieira, Machine Learning, Academic Press, 101-121, (2020).
https://doi.org/10.1016/B978-0-12-815739-8.00006-7 - Breiman, L. Random Forests. Machine Learning, 45:5–32, (2001). https://doi.org/10.1023/A:1010933404324
- Manning, Christopher D., Raghavan, Prabhakar, and Hinrich Schütze. "Introduction to Information Retrieval." Cambridge University Press, (2008).
https://nlp.stanford.edu/IR-book/pdf/irbookonlinereading.pdf - T. Cover and P. Hart, "Nearest neighbor pattern classification," in IEEE Transactions on Information Theory, vol. 13, no. 1, 21-27, January (1967).
https://ieeexplore.ieee.org/document/1053964 - Freund, Y., & Schapire, R. E., A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119-139, (1997).
https://doi.org/10.1006/jcss.1997.1504 - Chen, T., & Guestrin, C., XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785-794, (2016).
https://doi.org/10.1145/2939672.2939785 - Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A.. CatBoost: unbiased boosting with categorical features. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 6638-6648, (2018).
https://doi.org/10.48550/arXiv.1706.09516 - Python Software Foundation. pickle — Python object serialization. Python 3.10.7 documentation. Retrieved June 18, (2024).
https://docs.python.org/3/library/pickle.html - Abe, S. N. Lime. GitHub Repository. 2021. Retrieved June 18, (2024). https://github.com/marcotcr/lime
- Python Software Foundation. Python 3.10.9. Python Software Foundation. 2023. Retrieved June 18, (2024).
https://www.python.org/downloads/release/python-3109/ - Kluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B., Bussonnier, M., Frederic, J., Kelley, K., Hamrick, J., Grout, J., Corlay, S., Ivanov, P., Avila, D., Abdalla, S., Willing, C., & Jupyter Development Team. (2016). Jupyter Notebooks — a publishing format for reproducible computational workflows. Jupyter Project. Retrieved June 18, (2024). https://jupyter.org/
- American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed., American Psychiatric Publishing, (2013).
- World Health Organization. The ICD-10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines. World Health Organization, (1992).
- Spitzer RL., Kroenke K., Williams JBW. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. The Journal of the American Medical Association. 282(18);1737-1744. (1999). https://doi.org/10.1046/j.1525-1497.2001.016009606.x
- Beck AT. Depression: Causes and Treatment. Philadelphia: University of Pennsylvania Press. (1972).
- Hamilton MA., RATING SCALE FOR DEPRESSION, Journal of Neurology, Neurosurgery & Psychiatry. 23:56-62. (1960).https://doi.org/10.1136/jnnp.23.1.56
No competing interests reported.