Identifying Psychosis Episodes in Psychiatric Admission Notes via Rule-based Methods, Machine Learning, and Pre-Trained Language Models

Early and accurate diagnosis is crucial for effective treatment and improved outcomes, yet identifying psychotic episodes presents significant challenges due to its complex nature and the varied presentation of symptoms among individuals. One of the primary difficulties lies in the underreporting and underdiagnosis of psychosis, compounded by the stigma surrounding mental health and the individuals’ often diminished insight into their condition. Existing efforts leveraging Electronic Health Records (EHRs) to retrospectively identify psychosis typically rely on structured data, such as medical codes and patient demographics, which frequently lack essential information. Addressing these challenges, our study leverages Natural Language Processing (NLP) algorithms to analyze psychiatric admission notes for the diagnosis of psychosis, providing a detailed evaluation of rule-based algorithms, machine learning models, and pre-trained language models. Additionally, the study investigates the effectiveness of employing keywords to streamline extensive note data before training and evaluating the models. Analyzing 4,617 initial psychiatric admission notes (1,196 cases of psychosis versus 3,433 controls) from 2005 to 2019, we discovered that the XGBoost classifier employing Term Frequency-Inverse Document Frequency (TF-IDF) features derived from notes pre-selected by expert-curated keywords, attained the highest performance with an F1 score of 0.8881 (AUROC [95% CI]: 0.9725 [0.9717, 0.9733]). BlueBERT demonstrated comparable efficacy an F1 score of 0.8841 (AUROC [95% CI]: 0.97 [0.9580,0.9820]) on the same set of notes. Both models markedly outperformed traditional International Classification of Diseases (ICD) code-based detection methods from discharge summaries, which had an F1 score of 0.7608, thus improving the margin by 0.12. Furthermore, our findings indicate that keyword pre-selection markedly enhances the performance of both machine learning and pre-trained language models. This study illustrates the potential of NLP techniques to improve psychosis detection within admission notes and aims to serve as a foundational reference for future research on applying NLP for psychosis identification in EHR notes.


INTRODUCTION
Psychotic disorders are a category of mental disorders characterized by abnormal thoughts and perceptions and typically present with positive symptoms such as delusions and hallucinations.Accurate identi cation of patients experiencing psychosis is important for both clinical care and research.
Considerable evidence exists of an association between duration of untreated psychosis and clinical and functional outcomes. 1,2For example, a 12-year prospective study of 171 patients experiencing rstepisode psychosis found that at follow-up, patients who had experienced longer delays in initial treatment had, on average, poorer remission status, more severe symptoms, and greater social and quality and life impairment. 3However, psychosis case identi cation is complicated by the fact that it is often underreported and under-diagnosed. 4Individuals with psychosis often lack insight, leading to delays in seeking care.Moreover, on upon presentation for a rst psychiatric hospitalization for recent onset psychosis, patients are often guarded and withhold key information, complicating diagnosis.
6][7] Although emerging research proposes to use EHRs for disease detection and prediction, most existing works related to psychosis only use structured data such as ICD-10 codes, which can be inaccurate, vague, or missing entirely. 2,3,8,9Meanwhile, clinical notes, among the most reliable resources for obtaining related information given their rich context, have been less studied.
However, analyzing these data using traditional statistical methods can be time-consuming and may not capture the data's complexity and nuances.Previous research has demonstrated the potential of machine learning and deep learning techniques for accurately identifying patients with psychosis based on their structured EHR data. 10In a recent study conducted in South Korea, clinical data were extracted from EHRs from individuals with psychotic disorders in order to predict relapse.Three natural language processing (NLP)-enriched models were developed using three types of clinical notes (psychological tests, admission notes, and initial nursing assessment) and one complete model was developed using all three note types. 11Their results show that NLP models using clinical notes were more effective than models using only structured data, suggesting the importance of unstructured data in psychosis detection.In another study, NLP-derived variables of psychiatric symptoms and substance use were used to predict conversion to psychosis among patients with a prior psychiatric diagnosis. 12A study that combined clinical and temporal data from EHRs using a combination of rule-based information extraction and supervised machine learning methods predicted age of psychosis onset in a sample of individuals with a diagnosis of schizophrenia. 13However, none of these studies have utilized deep learning-based methods to analyze the EHR data despite a wealth of literature demonstrating that deep learning techniques have shown promise in clinical information studies.
One of the challenges of using deep learning techniques to analyze EHR data is the di culty of handling the exponentially increasing computing requirements in response to the input length.To address these challenges, one way is to mitigate the noise problem by shifting the prediction level from the patient to the note section or sentence. 14Such approach may improve the performance of deep learning models by providing more granular context, but they also exacerbate the labeling workload, which can be laborintensive and time-consuming.
In previous work, 15 we validated a keyword-assisted method that uses Term Frequency-Inverse Document Frequency (TF-IDF) and machine learning classi ers to identify related keywords and extract relevant sentences before feeding them to deep learning models for the identi cation of patient demographics.By focusing on sentences with relevant keywords, this method minimizes the amount of irrelevant information that is fed to the deep learning models and reduces the amount of data noise.In this study, we follow the same strategy to curate psychosis-related keyword lists for noise reduction and use deep learning-based methods to address the challenges of analyzing EHR data for identifying signs of newonset psychosis among patients hospitalized for psychiatric diagnoses.Our proposed approach has the potential to improve mental health care by identifying incident psychosis and facilitating the development of new methods for analyzing EHR data.This centralized process ensures admission notes are similar between patients with and without psychosis.Among 21,381 patients admitted to McLean between 2005 and 2019, we excluded 10,339 patients who were older than 35 years of age, as the onset of psychosis typically occurs in adolescence and young adulthood.To identify patients with rst hospitalization, we used a combination of structured data and text mining.Using structured data, we excluded patients with a previous hospitalization associated with an ICD diagnosis code for a psychiatric diagnosis within the MGB healthcare system.Figure 1 shows the process of study population identi cation.Future details about identi cation of psychosis cases are described in the next section.

Clinician identi cation of psychosis
Since this healthcare system is an open system and patients may have had an initial psychiatric hospitalization in a hospital outside of MGB, we extracted admission notes.Admission notes are divided into sections that contain history of present illness that incorporates information from patient and informants (e.g., family members), past psychiatric and medical history, family history of psychiatric illness, social and developmental history, current medications and past medication trials, ndings from a medical examination, and a formulation where differential diagnosis and precipitating factors are discussed, followed by a working diagnoses and initial treatment plan.We rst processed admission notes by using regular expressions to de-identify sensitive information and eliminate extraneous information in the notes, including address, dates, times, provider/hospital names, patient identi ers, and zip codes, along with numerical values containing more than three digits.Addresses were masked with "[address]", doctor names with "[doctor]", hospital names with "[hospital]", and dates with " [date]."Additionally, any information derived from templates and any duplicated text were systematically identi ed and excised from the dataset.Next, we applied text mining to identify patients with rst hospitalization: Python scripts were used that searched for keywords and surrounding contexts to extract phrases for each admission note that indicated rst or prior previous hospitalization and surrounding context (e.g., "multiple prior hospitalizations", "no prior psych hospitalizations") in past psychiatry section of note, which includes section on past hospitalizations.Phrases were manually reviewed to identify incident psychiatric hospitalizations.We excluded patients with central nervous system disease identi ed with ICD-9 and ICD-10 codes.We used text mining to exclude patients with past history of psychosis or mania by searching for terms and surrounding context to extract phrases consistent with new onset (e.g., " rst break") or past history of psychosis or mania (e.g., "patient with past history of schizophrenia").
After applying the above exclusion criteria, 4,629 patients remained in the nal sample.To classify patients as presenting with psychosis or another psychiatric disorder, text mining was used to identify terms related to symptoms and diagnosis of psychosis as well as words often used to describe patients with psychosis: hallucinations, delusions, psychosis/psychotic, schizo-, rst break, persecutory, voice, ideas of reference, referential, thought withdrawal, broadcasting, insertion, paranoia/paranoid, loose (loosening of associations), impoverish, magic (magical thinking), ight of ideas, grandiose/grandiosity, tangential, bizarre, strange, erratic, and/or odd.The terms were selected by psychiatrists with expertise in psychosis who provided clinical care on inpatient units and were familiar with McLean admission notes.For each patient, phrases for each instance of one of the keywords and surrounding context were extracted.Phrases were manually reviewed by two psychiatrists with expertise in psychosis (LVM and AKS), who rated as case with psychosis or control without psychosis, as well as con dence in rating (con dent, not con dent).The medical records associated with the admission were manually reviewed for all individuals where there were discordant ratings or rated by at least one rater as not con dent.A consensus meeting was held to determine the nal classi cation of discordant/not con dent ratings.To compare the rating of psychosis using these phrases with the gold standard of manual review of full medical records, the notes of 470 patients were randomly selected (~ 10% of the full sample).The positive predictive value (PPV) of phrase review to classify patients with and without psychosis compared with gold standard medical record review was 97%.
Demographic and clinical characteristics of the overall sample are provided in Table 1, comparing those diagnosed with psychosis (n = 1,196) to controls without psychosis (n = 3,433).Both groups had a similar median age of around 22 years, but the psychosis group featured a higher proportion of males (65.6% versus 47.7%).Racial composition varied, with the psychosis group having a higher representation of Black individuals (11.5% versus 6.9%) and a slightly lower percentage of White individuals (70.7% versus 75.5%).The psychosis group also showed a higher prevalence of public insurance users (17.6% versus 9.2%), indicating possible socio-economic differences.Substance use patterns differed notably between the groups, especially in cannabis and alcohol use, with the psychosis group showing higher daily cannabis use (28.2% versus 14.6%) and lower active alcohol use disorder (21.2% versus 27.5% in controls).a Signi cant difference between cases and controls p < 0.05.
b Cell counts ≤ 10 suppressed as privacy protection.
Figure 2 illustrates the study design and the intended application of this study.The goal is to assess and identify the most effective NLP model for detecting episodes of psychosis from psychiatric admission notes.Our approach is systematic, starting with the extraction of relevant information from the admission notes using pre-identi ed keywords.These keywords, rooted in prior research 16 and re ned by our expert consensus, served to sieve through the notes and retain content that was most indicative of psychosis.
Upon preparing the data, we conducted a comparative analysis of three mainstream NLP approaches.
The rst was a series of rule-based algorithms, which relied on a predetermined set of rules for identifying psychosis.The second approach entailed various machine learning models, which were trained on features extracted by TF-IDF to recognize patterns that might signify psychosis.Lastly, we evaluated pretrained language models, leveraging their extensive prior training on diverse language datasets, which we ne-tuned for our speci c task.

METHODS
Figure 2 illustrates the study design and the intended application of this study.The goal is to assess and identify the most effective NLP model for detecting episodes of psychosis from psychiatric admission notes.Our approach is systematic, starting with the extraction of relevant information from the admission notes using pre-identi ed keywords.These keywords, rooted in prior research 16 and re ned by our expert consensus, served to sieve through the notes and retain content that was most indicative of psychosis.
Upon preparing the data, we conducted a comparative analysis of three mainstream NLP approaches.
The rst was a series of rule-based algorithms, which relied on a predetermined set of rules for identifying psychosis.The second approach entailed various machine learning models, which were trained on features extracted by TF-IDF to recognize patterns that might signify psychosis.Lastly, we evaluated pretrained language models, leveraging their extensive prior training on diverse language datasets, which we ne-tuned for our speci c task.

Rule-Based Algorithms
Rule-based NLP algorithms are designed around a set of prede ned linguistic rules.These algorithms rely on speci c patterns, keywords, phrases, or a combination of linguistic markers that have been historically associated with psychosis.Examples of rule-based approaches that could be employed on our task include keyword identi cation, pattern matching, semantic rules, etc.
Keyword identi cation is the simplest NLP method that involves scanning the text for speci c keywords or phrases that are strongly associated with psychotic symptoms, such as "hallucinations," "delusions," "disorganized speech," or "catatonia."The presence of these keywords might trigger a ag for potential psychosis.Pattern matching typically uses regular expressions to nd patterns that suggest psychosis.It might look for complex patterns of speech that are indicative of disorganized thinking, a symptom of psychosis, like sentences that lack logical ow or abrupt topic changes.Semantic Rules involves creating rules that not only look for speci c words but also their semantic relations.For example, rules that link terms like "hears" with "voices" or "believes" with "being followed" could indicate hallucinations or paranoid delusions, respectively.
In this study, we have chosen to focus on keyword identi cation, acknowledging that rule-based approaches typically yield suboptimal performance in terms of generalizability.Our objective is to devise a method that maximizes generalization across studies.While both pattern matching and semantic rules could potentially enhance the model's precision, they necessitate an extensive review of the study corpus to develop bespoke rules.Such customization, however, tends to limit the applicability of the method to other studies and corpora, as the rules may not be universally applicable or effective in different contexts.Therefore, we prioritize broader applicability over corpus-speci c optimization in our methodological approach.
We adopted the keyword lists developed in a 2019 study by Viani, et al, 8 which trained various models on use-case speci c EHR texts from early psychosis intervention services, institution-speci c discharge summaries, and external clinical texts and also experimented with pre-trained embeddings from MEDLINE/PubMed.Their methodology encompassed diverse data sources, ensuring broad applicability and robustness in generating relevant terms, with the goal of developing an automated NLP model that could be applied to diverse settings outside of McLean Hospital.
The paper presented three keyword sets: the foundational 26 base seed terms related to psychosis symptoms and two generated lists segmented into unigrams and bigrams.Each term from the generated lists was manually categorized as a Relevant Term (RT), a Potentially Relevant Term (PT), or Not Relevant (NT).Guided by this classi cation, we formulated two distinct rules, which we refer to as "strict" and "broad", to construct our own keyword lists from the amalgamated unigram and bigram terms: Strict Rule: A term was considered strictly relevant only if it was unanimously identi ed as an RT by both annotators.
Broad Rule: A term was deemed broadly relevant if either (1) one annotator marked the term as an RT while the other marked it as a PT or NT, or (2) both annotators agreed on marking the term as a PT.
Following this, we integrated the terms ltered via these criteria with the original seed terms.During postprocessing, underscores were substituted with spaces, and plural forms were streamlined.To further reduce computational complexity, both lists were re ned to exclusively feature terms with the shortest unique substrings.We also made some minor changes in classi cation of terms from Viani et al., as annotators, who have clinical experience working with these notes at McLean Hospital, classi ed notes in accordance with their typical experience (e.g., moved the term "persecutory" from broad to strict because exclusively used in context of describing delusions).Machine learning algorithms are excellent at identifying patterns, but they require data to be in a format that they can process-essentially, numerical.Since textual information is inherently non-numeric, we must convert words into some form of numerical representation, or embeddings, that encapsulate the signi cance of the words within the context of the document.

Textual inputs to embeddings
There are several methods for encoding textual information, including one-hot encoding, word embeddings like Word2Vec, and the bag-of-words model.Each of these has its strengths, but they might not be the best t for clinical informatics due to various limitations, such as ignoring word order (bag-ofwords) or being computationally intensive (Word2Vec).
We chose the TF-IDF encoding because it provides a balance that is particularly advantageous in the psychiatric domain.TF-IDF quanti es the importance of a word in a collection of documents.It increases with the number of times a word appears in a document (Term Frequency) but is offset by the frequency of the word across all documents (Document Frequency).This means that common words across all documents are deemed less important, while unique words to a document are given more weight.
In our study, we took additional steps to re ne the text data for our machine learning models.We removed 'stopwords', which are common words like "the", "is", and "they" that offer little diagnostic value.We also limited the features to words that appear with a certain frequency-neither too common to be trivial nor too rare to be irrelevant.This was to ensure the words we used as features were statistically signi cant and had the potential to contribute meaningfully to the diagnosis of psychosis.Our feature set included unigrams (single words), bigrams (pairs of words), and trigrams (three-word phrases) to capture not just the signi cance of individual words but also the context provided by their adjacent terms.This is important because, in psychiatry, the context in which a word appears can be as telling as the word itself.More details of this process are provided in Appendix A.

Machine learning classi cation models
Upon transforming psychiatric admission notes into machine-readable embeddings, we used them to develop and evaluate a suite of machine learning classi ers.We included a diverse set of machine learning methods to ensure a comprehensive analysis.This diversity acknowledges that each model's unique strengths, assumptions, and potential biases play a signi cant role in its performance in detecting psychosis from textual data.By employing a range of models, we aim to ensure that our ndings are robust and not merely an artifact of a single algorithm's particular tendencies.In detail, we chose four classi ers that are renowned for their e cacy in clinical informatics: 1. Logistic Regression.Known for its simplicity and interpretability, estimating the probability of a binary outcome from input features, it operates under the assumption of a linear relationship between the features and the log odds of the outcome and presumes independence between features.However, this can be a potential bias if signi cant non-linear interactions exist within the psychiatric data, which logistic regression may fail to capture.2. Random Forest.As an ensemble of decision trees that can manage a large number of features, random forest classi ers are adept at classifying complex datasets.However, while they are less prone to over tting compared to individual decision trees and are good at capturing non-linear relationships, they can still be biased towards more frequent categories or features with more levels, which could overshadow the subtle patterns of psychosis in an unbalanced dataset.3. Multilayer Perceptron (MLP).As a type of neural network, it is valued for its capacity to learn complex functions through multiple layers of neurons.It assumes that intricate patterns can be discerned through these layers, which is useful for identifying nuanced language patterns in psychiatric notes indicative of psychosis.Nevertheless, biases can arise if the training data isn't comprehensive, potentially causing the MLP to overlook less common but clinically relevant expressions of psychosis.4. XGBoost.Leveraging a boosting algorithm that builds models sequentially to correct prior errors, XGBoost assumes that continuous learning from mistakes enhances performance.While known for its accuracy and e ciency, it can become biased by overemphasizing outliers or noisy data, and without careful tuning, it may over t, learning the training data too well and failing to generalize to new, unseen data.

Pre-trained Models
Pre-trained language models are a signi cant advancement in the eld of deep learning, which is a subset of machine learning focused on algorithms inspired by the structure and function of the brain called arti cial neural networks.The term "pre-trained" refers to the process where these models have already learned a substantial amount of English language understanding before they are ne-tuned for a speci c task-much like a medical student who has gone through years of training before specializing.
The pre-training equips these models with a deep knowledge of language structure and word relationships, allowing them to generate embeddings-numeric representations of text that capture semantic meaning.
In practice, these models serve primarily as sophisticated embedding generators that translate textual data into a numerical form, rather than acting directly as classi ers.The rich embeddings they create are versatile and can be utilized in a range of downstream tasks, prediction being just one example.Given the complexity and depth of understanding inherent in these pre-trained models, the classi ers used in conjunction with them can be relatively straightforward.A common choice is a linear classi cation layer, which, despite its simplicity, is su cient for making decisions based on the comprehensive information contained within the embeddings.
In this study, we chose Bidirectional Encoder Representations from Transformers (BERT) 17 models, which strike a balance between size, computational e ciency, and performance.While there are larger language models available, known for their robust performance in processing human languages, BERT's architecture offers a more practical alternative for clinical research environments.The latest large language models, though powerful, require considerable computational resources for ne-tuning and inference processes-resources that many clinical labs may not have.BERT models, being medium-sized, demand less in terms of computing power and data for training, which aligns better with the typical resource constraints found in clinical settings.
We utilized specialized BERT variants, namely ClinicalBERT 18 and BlueBERT, 19 which have been pretrained on extensive medical corpora, including MIMIC-III and PubMed.This pre-training imbues the models with an inherent understanding of medical terminology and documentation structure, allowing them to excel at identifying clinical conditions from text data, such as signs of psychosis in patient notes.
Their pre-existing familiarity with medical lexicon and semantic constructs positions them as e cient tools for parsing clinical notes, providing a signi cant advantage over models that have not been specialized, such as generic language models or those trained on non-medical text.
For both models, we implemented the architectures using all BERT embedding layers while freezing all but the last layer.To tailor the models for the task of psychosis identi cation, we added a linear classi cation layer.Fine-tuning was conducted with all but the last embedding layer and the added linear classi cation layer xed to avoid over tting.
Pre-trained language models like BERT use a self-attention mechanism which computes the attention scores for each pair of tokens (i.e., chunks of words) in the input sequence and thus requires computational resources that grow quadratically with the increase in input length.The two BERT models we use both set the maximum number of tokens per input.Therefore, for inputs with lengths more than 512 tokens, we truncate them to keep the rst 512 tokens.More experiment settings regarding these models can be found in Appendix C.

Training and evaluation details
As depicted in Fig. 3, the lengths of psychiatric admission notes are both long and variable, often surpassing the token limitation imposed by BERT models, and potentially introducing noise into machine learning models.Consequently, we applied the three-tiered keyword lists developed during the rule-based method phase as an initial ltering step before inputting data.
Prior to the experiments, the dataset was partitioned into training, validation, and test subsets.The training set's purpose is to train both machine learning and pre-trained language models, while the validation set is utilized to test the models during the training phase.This intermediate subset is crucial as it enables the model to iteratively make predictions and monitor the progression of the learning process.The test subset acts as a nal, consistent measure for validating the performance of all models.
We prepared three distinct versions of the dataset using the keyword lists: (1) the full notes, (2) notes compiled from sentences containing broadly relevant keywords, and (3) notes assembled from sentences with base relevant keywords.We omitted creating a set for strictly relevant keywords because every psychiatric admission note contained at least one base keyword.Thus, our comparison was between the base keywords (the most pertinent as determined by psychiatrists) and the broadly relevant keywords (expert-reviewed keywords identi ed by NLP methods).
For rule-based algorithms, which do not require training, evaluation was conducted directly on the test set.Machine learning algorithms, not constrained by input length, were assessed using the full notes as well as the sets ltered by broadly relevant and base keywords.Pre-trained language models, due to their limited input length capacity, were only applied to the sets of broadly relevant and base keywords, as truncating full notes to the rst 512 tokens could result in the loss of critical information.

Evaluation methods and metrics
We used the F1 score, the harmonic means of precision (positive predictive value [PPV]) and recall (sensitivity), as the primary metrics for evaluating our models' performance.We also reported the mean and standard deviation of sensitivity, speci city, PPV, negative predictive value (NPV), and accuracy.This allowed us to assess the stability and robustness of our results and determine the overall performance of the algorithms.
For the machine learning algorithms and pre-trained language models, we used a bootstrap resampling strategy with 1000 samples to estimate the con dence interval of the performance metrics.In addition, we report their area under the receiver operating characteristic curve (AUROC), area under the precisionrecall curve (AUPRC), and respective 95% con dence intervals (CI).

Non-NLP baseline: ICD code identi cation
used the Classi cation of Diseases (ICD) codes from discharge notes, which represented the principal diagnosis at discharge, as our baseline to compare NLP-based methods.Notably, discharge notes are expected to offer higher diagnostic accuracy compared to admission notes because they encompass the entirety of a patient's hospital stay, including all diagnoses, treatments, diagnostic clari cation, and outcomes.The rule-based algorithm is straightforward but highly speci c, employing an exact-match approach.When a patient's discharge notes include any ICD-9 or ICD-10 codes encompassing psychosis, the algorithm ags that patient as having psychosis.Conversely, if these codes are absent from the discharge notes, the patient is not considered to have psychosis.

RESULTS
Table 3 shows the comparative evaluation results.The most effective methods per metric are highlighted in green, and the least effective in red.The F1 score was employed as the primary metric to ascertain the optimal model or algorithm.
We observe that the non-NLP-baseline ICD identi cation exhibited reasonable performance, with an F1 score of 0.761, a recall of 0.714, and an NPV of 0.905.While its speci city and accuracy were relatively high at 0.944 and 0.884, respectively, it lagged in precision at 0.814.
The keyword-matching algorithms, particularly those based on all keyword lists, demonstrated exceptionally high recall and NPV scores, both reaching the maximum value of 1.0 at the cost of extremely low precision (around 0.26) and speci city (as low as 0.0064).Nonetheless, the keywords served as great pre-selection methods, as we observe that is that most of the models had performance gains going from full notes to base-keyword identi ed notes.Appendix D shows summary statistics of keyword distribution in notes of cases of psychosis and controls.
The machine learning methods, TF-IDF + XGBoost demonstrated the best performance, achieving F1 scores of 0.8646, 0.8787, and 0.8881 on the three sets of notes, respectively.The model also recorded the highest accuracy of 0.9330, 0.9385, and 0.9427 on the full and selected sets.This consistent outperformance of other models underscores XGBoost's robustness across different feature spaces and its ability to balance precision and recall effectively.
One noteworthy observation relates to the performance of the logistic regression model, particularly in terms of precision.On the full dataset, the logistic regression model achieved a precision of 0.9365 and a speci city of 0.9841, the highest among all classi ers in the category using full notes.Nonetheless, logistic regression models were low in recall (sensitivity) scores (0.6741).
Both BlueBERT and ClinicalBERT demonstrated high AUROC and AUPRC, re ecting their capabilities in exceptional class differentiation and classi cation.However, their AUROC and AUPRC are subject to greater uctuation in 95% con dence intervals compared to XGBoost.
As of our key metric of F1 score, BlueBERT trained and tested on admission notes pre-selected using only the base keywords achieves a remarkable F1 score of 0.8841.This surpasses the version trained and tested on admission notes pre-selected using all available keywords, which posts an F1 score of 0.8591.BlueBERT trained on base keyword identi ed notes demonstrates stronger precision and speci city, reinforcing the idea that careful keyword selection helps reduce noise and enhance model performance.

5.
Our study comprehensively evaluated NLP methods in identifying psychosis in psychiatric patients through a multi-pronged analysis involving rule-based algorithms, machine learning, and pre-trained language models using psychiatric admission notes from 4,629 patients.We found that keyword preselection of notes increased classi er performance, XGBoost using TF-IDF-encodings performed the best among all methods, BlueBERT offered similar performance.A previous study 12 that predicted conversion to psychosis among patients with a psychiatric disorder identi ed 14 terms that predicted psychosis, which included non-speci c symptoms of psychiatric illness (e.g., insomnia), substance use (e.g., cannabis and cocaine use), and symptoms more speci c to psychosis (e.g., delusions).Symptoms more relevant to psychosis, speci cally delusions, agitation, and paranoia, were the strongest predictors of psychosis.This study used supervised machine learning (support vector machine) based on a lexicon of keywords indicating symptoms of serious mental illness.Our study adds to this research by showing that deep learning methods can enhance the performance of models to identify psychosis among patients admitted to a psychiatric hospital from electronic health records.
In our evaluation, keyword-based algorithms, especially those using a less curated set of keywords, demonstrated extremely unbalanced performance metrics.These algorithms achieved essentially perfect recall but had very poor precision (i.e., positive predictive value), as low as 0.2587.Such results indicate that while the keyword-based algorithms are adept at identifying relevant cases, they also produce a high number of false positives.A signi cant drawback of such algorithms lies in their inability to grasp context, particularly when it comes to negations and expressions of uncertainty; phrases like "no signs of psychosis" can trigger false positive results.This de ciency in contextual comprehension underscores the necessity for more sophisticated NLP techniques or machine learning models capable of grasping the semantics of clinical language.
Nonetheless, the base keywords played a pivotal role in pre-selecting relevant information from clinical admission notes, thereby streamlining the performance of both machine learning classi ers and pretrained language models.By ltering out extraneous details, the keyword-based selection process not only made the classi ers more e cient but also optimized pre-trained language models to work within input size constraints.This dual utility enhanced diagnostic accuracy and computational e ciency across the board.Interestingly, however, we discovered that incorporating additional relevant keywords extracted from extensive PubMed texts and clinical notes using word embedding techniques did not lead to further improvements in the predictive models' performance.This discovery may be attributable to the fact that, with the foundational keyword lists that include more directly related keywords, the matched text already reached the input limit.As our cohort was identi ed in a psychiatric hospital with high rates of psychosis, applying our model to the general population may be more challenging due to rarity of symptoms of psychosis.However, as the keyword lists adopted from Viani et al. 16 used diverse data sources, this comprehensive list of keywords may have applicability in other settings.In addition, many of the signs and symptoms of psychosis are domain-speci c, such that relevant terms were curated by domain experts using their clinical experience.
Among the machine learning classi ers we tested, XGBoost emerged as the most promising, achieving an F1 score of 0.8881.Remarkably, the best-performing pre-trained language model exceeded all other models, boasting an F1 score of 0.8841.This represents an increase from the ICD codes' F1 score of 0.7608, which is noteworthy since ICD codes re ected diagnosis at discharge and may be more accurate as they incorporate information gained from observation and clinical evaluation over the course of days to weeks.The use of keyword-based selection in tandem with machine learning and pre-trained language models offers a particularly promising avenue for identifying speci c patient cohorts in neuropsychiatric epidemiological studies.Traditional methods of cohort identi cation often rely on manual curation and are thus time-consuming and subject to human error.The method used for cohort identi cation for this study by clinicians, which included review of phrases related to psychosis and manual review of a subset of medical records, was labor-intensive and would not be feasible for larger samples.The NLP techniques presented in this paper could automate and signi cantly expedite this process, allowing researchers to rapidly and accurately isolate groups of patients who share certain diagnostic features, treatment histories, or risk factors.This level of granularity is crucial for epidemiological studies that aim to understand the complex interplay of various factors in neuropsychiatric conditions.For example, researchers could more easily identify cohorts for longitudinal studies on treatment e cacy or for crosssectional studies aimed at identifying environmental or genetic risk factors.Overall, the technological advancements discussed in the paper could revolutionize the way patient cohorts are identi ed and analyzed, thereby increasing the speed and accuracy of neuropsychiatric research.Automation of detection of psychosis in EHR notes could be useful clinically to identify patients with emerging psychosis, which currently relies on patient-initiated help-seeking 8 and speci c referral pathways for people at risk for suspected psychosis. 9Studies have shown that these current detection strategies are highly ine cient and unreliable, with only 5-12% of individuals at clinical high risk for psychosis (CHR-P) actually converting to rst-episode psychosis. 20Our methods could allow for detection of at-risk individuals across a broad hospital system, allowing triage to more intensive level of care or refer to clinics that provide comprehensive services for individuals with rst-episode psychosis.
However, the use of admission notes as the primary data source also has limitations.These notes are subject to clinician bias and incomplete information about patients.For example, patients may be too agitated, sedated from emergent medications or lack insight into symptoms to provide accurate information.Algorithmic bias may be introduced by clinician bias, as previous studies have shown that Black individuals with depression are more likely to be misdiagnosed with psychosis than White patients. 21Additionally, the absence of objective, quanti able biomarkers for mental health conditions further complicates diagnostic accuracy.Signs of mental health conditions may not be immediately evident and may only emerge over time, affecting the classi ers' performance, particularly in terms of speci city and precision.Therefore, these models should be part of a broader diagnostic toolkit, possibly incorporating longitudinal data, in-depth interviews, and other objective measures for a more comprehensive understanding of a patient's mental health.
Ethical considerations also arise, particularly regarding the potential for overdiagnosis or misdiagnosis, given the limitations of both keyword-based and machine-learning algorithms.Such risks necessitate rigorous validation and possibly the inclusion of human oversight in the diagnostic process.
In conclusion, while machine learning algorithms offer promise for improving mental health diagnosis, further research is required to address their limitations and to explore their integration into a more comprehensive diagnostic framework.We recommend that future studies consider incorporating more diverse and longitudinal data sources to validate and potentially improve upon our ndings.

CONCLUSION
This study presents a comprehensive evaluation of common NLP techniques for identifying psychosis in from psychiatric admission notes, highlighting the potential of keyword pre-selection and advanced algorithms to re ne diagnosis from EHR data.It serves as a guidebook for future studies in using NLP to identify psychosis from EHR.This study also underscores the need for further research to optimize these NLP approaches, aiming for their integration into a holistic diagnostic framework that can augment the capabilities of mental health professionals.Study Design and Intended Application.The objective of this study is to evaluate and compare the effectiveness of rules-based algorithms, machine learning models, and pre-trained language models in detecting episodes of psychosis from psychiatric admission notes of patients.To address the input length limitations of pre-trained language models and minimize extraneous information, the study incorporates a note pre-selection process.This process involves the use of various keyword sets derived from existing literature, which have been reclassi ed and reviewed by experts for relevance and accuracy.

2 .
DATA SOURCES AND PREPARATION 2.1.Clinical Setting and Data Collection This study was conducted at McLean Hospital, a psychiatric hospital in Belmont, Massachusetts, and a member of the Mass General Brigham (MGB) integrated healthcare system.All study activities were conducted with the approval of the MGB Human Research Committee (IRB) with a waiver of informed consent according to 54 U.S. Code of Federal Regulations 46.116.The purpose of this study was to identify patients admitted to McLean for an initial psychiatric hospitalization between 2005 and 2019 and to classify patients as new onset psychosis versus another psychiatric disorder.All patients admitted to McLean are initially seen in a Clinical Evaluation Center, where they undergo a comprehensive psychiatric evaluation that is documented in the Admission Note.

Figures
Figures

Figure 1 Data
Figure 1

Table 1
Demographic and Clinical Characteristics of Cases and Controls.

Table 2
Psychosis-related keyword lists generated from previous research.

Table 3
is available in the Supplementary Files section.