Existing stoke prediction models
We chose the problem of predicting stroke in patients with atrial fibrillation as it has been well studied and is one of the only prediction problems to have been extensively validated. Therefore, we have ample benchmarks to compare to this study’s validations. The existing models we replicated were ATRIA, CHADS2, CHA2DS2-VASc, Framingham and Q-Stroke.
The Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) [3] model was developed on a cohort of 7,284 patients who were 18+ and had an atrial fibrillation outpatient diagnosis during 1997 or 1998 and internally validated on 3,643 patient hold out set obtaining an AUROC of 0.72. In the same paper, the authors also externally validated the model on a cohort of 33,247 patients aged 21+ with inpatient or outpatient atrial fib or flutter during 2006-2009, obtaining an AUROC of 0.7. The Congestive heart failure, Hypertension, Age > 75, Diabetes, prior Stroke/transient ischemic attack (CHADS2) score [5] was developed by combining two other stroke prediction models (using the variables from these models and assigning points) and was validated on 1,733 patients aged 65 to 95 years who had nonrheumatic AF. The CHADS2 score obtained an AUROC of 0.81 on this population. The CHA2DS2-VASc score [6] is another score based model that was developed using knowledge of risk factors. The model was validated on a cohort of 1,577 patients who were 18+ and had AF during 2003 to 2004 from 35 countries. The model obtained an AUROC of 0.61 for this patient population. The Framingham score [4] model was based on a Cox model developed using data from 705 patients aged 55 to 94 with initial atrial fibrillation. The internal validation, using a bootstrap approach, showed an AUROC of 0.66. The Q-Stroke [7] model was developed using primary care data from the UK consisting of 3, 549, 478 patients aged 25-84 with no prior stroke or anticoagulation use (except aspirin) and internally validated on 1, 897, 168 similar patients. When applying the model to predict the 10-year risk of stroke in female patients with AF at baseline, the AUROC was 0.65.
The existing models include a small number of variables, Table 1 summarizes the variables included in each model. Some of the variables are unlikely to be available in claims data and these are highlighted in red. A large number of Q-Stroke variables are not commonly found in claims data (or are UK specific), so this model is difficult to replicate in external non-UK databases. Although the internal validation for some of the models was as high as 0.8, previous external validation of the models tends to achieve an AUROC between 0.6 and 0.7.
The complete definitions for each variable (sets of SNOMED CT or RXNorm codes) are provided in Appendix A.
Prediction task
Within a target population of female patients with newly diagnosed atrial fibrillation and no prior stroke predict who will develop a stroke 1 to 365 days after initial diagnosis of atrial fibrillation.
Target populations definitions
The existing models were applied to two target populations. Both target populations consisted of female patients newly diagnosed with atrial fibrillation and no prior stroke or anticoagulant use but target population 1 was patients ages 65 or older and target population 2 was all ages.
Target population 1: The target populations was defined as females aged 65-95 with either:
- 2 atrial fibrillation records
- 1 atrial fibrillation in an inpatient setting
- 1 atrial fibrillation with an ECG within 30 days prior
and at least 730 days prior database observation and no prior stroke and no prior anticoagulant.
Target population 2: The target populations was defined as females with either:
- 2 atrial fibrillation records
- 1 atrial fibrillation in an inpatient setting
- 1 atrial fibrillation with an ECG within 30 days prior
and at least 730 days prior database observation and no prior stroke and no prior anticoagulant.
Outcome definition
The stroke outcome was defined as:
- An ischemic or hemorrhagic stroke recorded with an inpatient or ER visit
The code sets used to define atrial fibrillation, ECG and ischemic or hemorrhagic stroke are presented in Appendix B. The full analysis code (data creation and model evaluation) is available at: https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation
Time-at-risk
We predicted stroke occurring 1 day until 365 days after the initial atrial fibrillation start date.
Sensitivity analysis
As using anticoagulants can impact a patient’s risk of stroke, in addition to the two target populations we did a sensitivity analysis where we removed people who had an anticoagulant prescription record that may have intervened in the stroke development. In the additional sensitivity target populations we censored patients at the point an anticoagulant was recorded, so any patient with an anticoagulant during the time-at-risk period was effectively removed from the target population unless they had a stroke prior to the anticoagulant,
Statistical analysis
The prediction model performances were evaluated using the area under the receiver operating characteristic (AUROC) curve. Discriminative measure and confidence intervals were also calculated when the data are small.
Datasets
The datasets used to evaluate the models are:
IBM MarketScan® Commercial Database (CCAE) is a United States employer-sponsored insurance health plans claims database. The database contains claims (e.g. inpatient, outpatient, and outpatient pharmacy) from private healthcare coverage to employees, their spouses, and dependents, so patients are aged 65 or younger.
IBM MarketScan® Medicare Supplemental Database (MDCR) represents health services of retirees in the United States with primary or Medicare supplemental coverage through privately insured fee-for-service, point-of-service, or capitated health plans. The patients are aged 65 or older.
IBM MarketScan® Multi-State Medicaid Database (MDCD) contains adjudicated US health insurance claims for Medicaid enrollees from multiple states and includes hospital discharge diagnoses, outpatient diagnoses and procedures, and outpatient pharmacy claims as well as ethnicity.
Optum© De-Identified Clinformatics® Data Mart Database – Socio-Economic Status (Optum Claims) is an adjudicated administrative health claims database for members with private health insurance. The population is primarily representative of US commercial claims patients (0-65 years old) with some Medicare (65+ years old) however ages are capped at 90 years.
Optum© de-identified Electronic Health Record Dataset (Optum EHR) is a US electron health record containing clinical information, inclusive of prescriptions as prescribed and administered, lab results, vital signs, body measurements, diagnoses, procedures, and information derived from clinical Notes using Natural Language Processing (NLP).
Stanford Translational Research Integrated Database Environment (STRIDE) is a clinical data warehouse that supports clinical and translational research at Stanford University. This resource includes the EHR data of approximately 2 million adult and pediatric patients cared for at either the Stanford Hospital or the Lucile Packard Children’s hospital. This study was completed on an OMP-CDM adherent instance of STRIDE.
Columbia University Medical Center’s (CUMC) OMOP CDM data come from New York Presbyterian hospital’s clinical data warehouse. The database comprises EHR data on approximately 5 million patients and includes information such as diagnoses, procedures, lab measurements and prescriptions.
Ajou Univeresity School Of Medicine (AUSOM) is a database in the format of OMOP-CDM version 5 for entire EHR data from 1994 to 2018 of Korean tertiary hospital, Ajou university hospital. It contains medical record of about 2.9 million patients.
The Integrated Primary Care Information (IPCI) is an electronic health care databases containing patients of Dutch general practitioners.
Each site had institutional review board approval for the analysis approval for the analysis, or used deidentified data and thus the analysis was determined not to be human subjects research and informed consent was not deemed necessary at any site.