Our study collected data from three academic institutions and one large accountable dental care organization that all used the same EHR system (Exan, Coquitlam, BC, Canada). Our research team, comprised of clinicians, informaticians, public health dentists, and statisticians, developed the following measure to assess and document periodontal disease risk and diagnosis.
Study population: We designed our measures while considering existing practice guidelines regarding assessing periodontal risks. The denominator of the measure included patients 16 years of age or older and had at least one completed or in progress comprehensive/periodic/or periodontal exam (D0120/D0150/D0180) in the reporting year (Figure 1a). The numerator included patients who had a completed periodontal probing charting, an assessment of all three periodontal disease risk factors (diabetes status, tobacco use, and oral home care compliance), and a periodontal diagnosis within six weeks of their comprehensive/periodic/or periodontal exam (Figure 1b).
Approach for Testing and Validating the Automated Query
Data from the 2015 calendar year was used for testing and validation purposes using the following steps:
Step 1:Measure automation (automated query): All test sites generated the sampling frame from their EHR using the same Structured Query Language (SQL) script, assuring all patients who were eligible would be included in both the denominator and the numerator.
Step 2: Sample size estimation: We estimated the sample size using the proportion of patients who received periodontal charting, periodontal risk assessment, and periodontal diagnoses for each site during the reporting period as derived from the automated query. We calculated the required sample size for a manual review with a precision of 5% around the expected effect size at the 95% confidence interval (CI) level.
Step 3:MeasureValidation: We validated the automated query performance through a manual chart review, which served as the gold standard. In our earlier studies,(13, 14, 33, 34) we effectively calibrated two trained reviewers at each site, by calculating the interrater reliability using 50 manual chart audits. When both reviewers achieved >80% agreement,(35) we proceeded with single reviews to complete the remaining charts, calculated the sensitivity, specificity, positive predictive value, and negative predictive value of the automated query.
Step 4:Measure score: EHR-measure proportions were calculated as a percentage of numerator/denominator for each site.
Step 5:Statistical Analyses: By site, descriptive analysis was employed for all measure scores. The frequencies and percent for the total number of patients who received tobacco, homecare (defined as the presence of heavy plaque), and diabetes screenings, had comprehensive periodontal charting and received periodontal diagnoses were calculated. Line graphs were generated to show site variations of the measure scores over time, and bar charts were generated to show the site distributions. In order to determine whether there were statistically significant differences between the measures scores calculated by manual review and those calculated by the query results, an independent sample hypothesis test was performed. All tests were conducted at the standard significance level of 0.05 (a = 0.05) and all analysis used Stata Statistical software release 14 for StataCorp LP. After the validation process, the measures were run again for three additional years (2016, 2017, and 2018).