We used data from two primary care centres with a list size of 6000 and 4000 patients in London to test and train the models. The raw data of coded histories were extracted from the clinical IT system supplier (EMIS) and imported into a secure local SQL database. Patient data was anonymised to ensure further data privacy to comply with GDPR. All identifiers were randomised, and only the GP practice with access to patient identifiable information for reviewing patient records can access data in the usual way.
Initial data exploration and interpretation conducted helped identify data quality issues. For example, each coded entry had two dates; the event date related to the date of the occurrence of the event and the audit date related to the date the code was added. These are often the same but different when retrospective entries are made, indicating that medical codes could have been altered at different time points.
Feature Set:
Asthma is a chronic respiratory disease related to inflammation of the airways. Common asthma symptoms include chest tightness, wheeze, cough and breathlessness. When these symptoms significantly compromise the airway, it can constitute the individual having an asthma exacerbation.
The input data contains registered patients' demographics and events, including diagnosis, findings, observations, and communication information. Prescriptions contain data about medications prescribed to the patients. Prescription data goes back five years, and events data goes back more than 15 years before the study. The following features were extracted from the patients' electronic health records, as shown in S1.
Some common asthma symptoms are cough, wheezing, sleep disorder, shortness of breath and chest tightness (13). These symptoms may occur in isolation or as a combination (8). Cough/ wheezing symptoms are considered bouts of a single episode if they occur within eight weeks. So, the first time a patient visits the surgery for cough or wheeze, it is considered an event, and subsequent visits are put together as a single episode if they fall within an 8-week window. Patients are more likely to have asthma when wheezing and atopic conditions occur. Atopic eczema is a condition that makes the skin red and itchy (14). According to Ellina Gillani et al, atopic dermatitis is considered an entry point for respiratory conditions such as asthma. They suggest that careful management of atopic dermatitis is essential to prevent the development of respiratory allergy or to reduce the severity of asthma and allergic rhinitis, another factor closely associated with asthma. According to Bergeron and Hamid (15), who conducted a study on asthma patients and patients with rhinitis, 40% of patients with allergic rhinitis have asthma, and 94% of patients with allergic asthma have rhinitis. They also state that the incidence of asthma and rhinitis increases with age and have found that keeping a check on rhinitis helps manage asthma better.
Allergens that can cause inflammation in the body can be identified using the immunoglobulins E values. A value of more than 0.35kU/L is considered elevated. Bronchodilator reversibility is a lung function test performed using a spirometer and a bronchodilator. A positive result is considered a strong indication of asthma. Spirometry is conducted to test the restrictiveness or obstructiveness of the lungs. Obstructive lung function can be found in asthma patients who have difficulty exhaling air. A positive result is when the ratio FeV1/FVC is less than 70%. Another feature considered is the frequency of asthma medications administered in a year to patients in the past; in this study, a period of 5 years is considered.
Evaluation:
In this study, several deep learning models – Multi-perceptron Layer (MLP), Convolutional neural networks (CNN), as well as linear models -- Logistic regression (LR), Random Forest (RF), Naïve Bayes (NB), Support vector machines (SVM), Linear ensemble model (ensemble) - were explored to analyse patient data. Both linear and deep models are effective in different ML applications, as seen in (16–22). The problem was a binary classification where patients were required to be classified as asthmatic or non-asthmatic. All models were provided with the same input features so that the outputs could be compared on an equal footing. The metrics used to evaluate the models are accuracy, specificity, sensitivity, AUC-ROC score, and positive predictive value (PPV). Accuracy is defined as the number of correct predictions for all predictions. Specificity measures how well the model identifies asthma patients, and sensitivity measures how well the model identifies negative samples or non-asthma patients. AUC score is used as the metric that indicates how well the model can distinguish between the classes. The higher the AUC score, the better the model's performance distinguishing between the positive and negative classes. As the models are being compared against an existing search-based system that identifies potential asthmatic patients, PPV was used as the appropriate metric, which helps indicate how many patients were correctly predicted as asthmatic. The output of the searches was a set of suspected asthma patients, who GPs reviewed before confirming their asthma status. The number of positive cases obtained from the reviews was compared against the predictions of the models.
The input features were extracted using several SQL queries with appropriate cut-off dates and other constraints and merged in the python environment. The 'cut-off' dates ensure that the model only sees the data before diagnosis and is not biased by subsequent codes assigned after diagnosis. In a previous review conducted between 2017 and 2019, 171 patients who were identified as suspected asthma patients were found using Smart SearchesTM. These searches are a set of intelligent queries built to filter out asthma suspected patients. The queries use asthma-related diagnoses and prescription codes to filter out patients. Of approximately 10,000 patients from 2 practices, 171 were suspected of having asthma. After the review of the case notes by GPs, 104 were identified as asthmatic. From this group of 171, only 114 patients EHR were currently available for the study, of which 66 were confirmed asthmatic and 48 non-asthmatic.
All diagnosis was confirmed after a rigorous review by the GP and the following spirometry where required. The review process occurred between July 2017 and May 2019. Audited patients obtained by the search-based system and reviewed by GPs make up the audited dataset, and non-audited patients were the rest of the patients from the practice. This audited dataset was used as the validation hold-out set for evaluating the models in the current study. Initially, data from one practice was used to train the models. For the non-audited patients' dataset, only about 6% (~410) of the patients on the asthma register are positive samples, and the rest, 94% (~5800) are non-asthmatic patients or negative samples. The reviewed audit dataset of 91 patients from the same practice comprises a reliable training set. Each suspected asthma patient found by the originally structured algorithm searches (Smart SearchesTM) has verified their asthma status by a GP. Unfortunately, as shown by experiments 1, 2 and 3 in Table 3, this reliable dataset was too small to be used as an effective training set. Therefore, the complete set of patients at the GP surgery (excluding a small test set) was used as a training set, with each patient's asthma status recorded in the GP surgeries asthma register used to label the samples.
The asthma register is an administrative ledger that follows the QOF guidelines and contains a list of patients diagnosed with asthma and who have had asthma medications prescribed to them in the last 12 months . Any patient diagnosed with asthma and receives a prescription remains on the asthma register, whereas patients without a prescription in the last 12 months are removed from the register. The asthma diagnosis and medication codes underlying the asthma register are the QOF cluster codes. These codes are assigned to patients to indicate a diagnosis, finding, treatment or procedure and is summarised in the Table 2 below. The asthma codes were provided to the model as required for the analysis. AST_COD cluster indicates asthma diagnosis and related codes, and ASTTRT_COD are codes assigned to the asthma medication/ treatment the patients receive. For all the experiments, ASTSPIR_COD codes were used to indicate if the spirometry test was conducted and the test results recorded.
<Table 1>
All the experiments were run with the validation split varied between 20% and 70% to ensure maximum exposure of the training data to the model. The models were first trained with the existing training set before applying the Synthetic Minority Oversampling Technique (SMOTE) to the train set to reduce the class imbalance. The models achieved some accuracy with the original dataset but learned better with the balanced training data. Class weights also helped with improving the predictions. To understand the predictability of the models, explainability techniques were used. Shapely additive explanations (SHAP) are useful to demystify a model, as shown in (23). In addition, they help to understand the 'why' of a decision made by the machine learning models. The SHAP values calculated from models help understand the impact of a feature on the model's decision on a patient. For example, with SHAP values, we could find that one of the features is valid for a few patients; otherwise, 0 was given a high priority by the models. After converting this feature to a categorical variable and retraining, the models showed a slight improvement in their accuracy. The summary plot of SHAP shows the importance of features on average, as shown in Figure 4.
Data from another GP surgery helped to increase the patient list size to 9000 and the number of patients on the asthma register to around 600. The models were tested and retrained with the new data.