To our knowledge, this is the first study that compared the value of methods to create a sample of women at high maternal risk in administrative data. In this analysis, the experimental sample created with the OCI using a cut-off value of 4 had a positive net benefit. This appears to be due to the high specificity of the weighted scoring of the OCI. The conventionally derived sample had a negative net benefit, which means it was less valuable than creating a sample by identifying no woman as having high maternal risk. In the case of a research sample, a negative net benefit means it is likely that the harm due to misclassification of women as having high maternal risk outweighed the benefit of correctly identifying women at high maternal risk. Misclassification bias would skew the result of any study toward no difference, and therefore increase the risk of a Type II error.
Though the conventional method of sample selection had better sensitivity, the low specificity of this methods resulted in high proportions of women classified at high maternal risk. Given that only 0.5% of women in these data experienced a poor maternal outcome, identification of 35% of the women as being at high maternal risk is likely the result of overestimation of risk status. Though the women identified with these methods may have had an elevated risk, it is unlikely such high proportions of women would be transferred to the highest acuity care to prevent maternal complications. It is more likely most of these women represent a category of moderate maternal risk rather than high risk.
Categorization as women at moderate maternal risk may better simulate clinical reasoning about care than a dichotomous categorization as either low or high risk. Women at moderate maternal risk likely received a higher level of surveillance with their obstetrician rather than being transferred to a sub-specialty practice. In these data, the OCI identified 190,672 (35%) women who might be considered moderate risk; that is, they had a comorbid condition but did not meet the threshold for high maternal risk identification. Stratification of administrative data into low, moderate, and high maternal risk provides an opportunity to better understand the implications of health system level interventions to prevent maternal morbidity and mortality.
In these data, the conventional model identified 35% of the sample as being at high maternal risk but captured 60% of the poor maternal outcomes. This is concerning as it suggests 40% of the severe maternal morbidity and mortality occurred in women with no coded ICD risk factors. It is beyond the ability of this study to determine if risk factors were identified by clinicians but not coded in the hospital discharge record. Additional work is needed to improve identification of maternal risk factors in administrative data.
The methods tested in this paper rely on ICD-9-CM codes, which may not reflect the true distribution of comorbid conditions in the community. Additionally, ICD-9-CM codes are an imprecise representation of the clinical condition of a patient. For example, administrative data records only include that gestational diabetes was present, not if it was well controlled. The sensitivity and specificity obtained in this study should not be generalized to the accuracy of the obstetric comorbidity index for control of confounding or in clinical use.
This study was delimited to risk identification methods that relied on comorbid conditions already known to increase maternal risk. It is possible another risk identification model not tested in this study, such as the Pregnancy Risk Score System, is superior to the OCI as a sample selection tool.5 It is also possible that other criteria, not included in the risk identification models used, would create a superior model.
To use the OCI as a sample selection tool rather than to control for confounding, a cut-off value was selected. The cut-off was selected using net benefit analysis to ensure the comparison of the value of the index was not hindered by assigning a random cut-off. Though the cut-off of 4 was superior for these data, this cut off is unlikely to be generalizable. These data were limited to singleton deliveries and appeared to include underreporting of some outcomes such as alcohol abuse. Such underreporting of alcohol abuse has been reported previously, which suggests this variable may be an inherent limitation to use of the index in administrative data.10 Researchers using sample selection with the OCI should assess the best cut-off value for the study data and report the cut-off used and rationale until a standard is determined.
Finally, this study compared methods of risk identification by assessing the accuracy of the method at identifying women who had a poor maternal outcome. However, the goal of identifying women at high maternal risk is to ensure they receive the necessary care to prevent poor maternal outcomes. This study is not able to account for the care women did or did not receive that, if controlled, may alter the value of the models. This limitation was considered acceptable because these data represent clinically identified risk and the associated outcomes with the available medical practice.
The present findings highlight the usefulness of the OCI as a tool to select a sample of women at high maternal risk in administrative data. The OCI was superior to the conventional practice of identifying women with any comorbid condition. Additionally, the OCI allowed identification of a group of women at moderate maternal risk; that is women with identified risk factors but not likely to need transfer to the highest acuity care to prevent poor maternal outcomes. This improved stratification of risk can prevent misclassification bias. Researchers who use the OCI should be aware of the potential for underreported conditions and therefore select the cut-off based on the data being analyzed.
This analysis has shown that sensitivity is low when applying conventional definitions of high maternal risk in administrative data. This is an important finding as it may indicate either the current understanding of maternal risk or the current recording of comorbid conditions in administrative data are not precise enough to create a standard research definition that will prevent misclassification bias during sample selection. In addition to the lack of clinical specificity, the variation in rate of false positives and false negatives for ICD codes during delivery admissions provide additional challenges to using administrative data.17 There are likely non-random reasons for both missing and mis-coded ICD codes that may hinder interpretation of findings when risk is measured by ICD coding.
The conventional method of maternal risk stratification grew from improved understanding of characteristics associated with poor maternal outcomes. These characteristics are classified as “risk factors” due to increase in relative risk and used to define high maternal risk in administrative data without consideration of the absolute risk. When applying definitions of maternal risk to sample selection in administrative data, understanding how inclusion of additional criteria decreases specificity, accuracy, and net benefit of maternal risk identification is important to preventing misclassification bias. Future research should continue to compare the usefulness of maternal risk stratification methods for administrative data, comparing net benefit of the methods in different study populations and evaluating the impact of misclassification of risk on estimation of benefit for population level interventions.