This systematic analysis of LOE supporting AAO PPP guidelines evaluated the proportion of guidelines with a LOE listed, as well as changes in reporting patterns over time and across specialties. Overall, this study demonstrated that while current guidelines report LOE for substantially fewer recommendations, a much higher proportion of recommendations are supported by evidence from randomized controlled trials. Eighty-eight percent of current recommendations with reported LOE had LOE I. Subgroup analysis by subspecialty showed similar trends. These results suggest that while current AAO PPPs emphasize evidence from randomized controlled trials, LOE from other types of studies may not be formally rated or reported.
Although analyses of guidelines have been performed in other specialties, such as cardiology [1, 2], prior investigation of evidence supporting guidelines in ophthalmology is limited. A 2015 study examined the LOE of papers published in four major ophthalmology journals and concluded that lower LOE publications would continue to play a large role in guiding the field of ophthalmology [3]. At first glance, the findings from our study do not appear to suggest this same trend among the reported LOE supporting AAO PPPs, as the vast majority of recommendations with reported LOE had the highest level of evidence.
However, our comprehensive review of the 2021 Cataract in the Adult Eye PPP and independent rating of LOE of the citations show that 30% of recommendations rely on level II and III evidence (vs. 18% level I), but the LOE was simply not reported in the PPP. The majority (52%) of recommendations did not have any citations, consistent with a prior study investigating the relationship between findings from systematic reviews and the 2015 AAO PPP on interventions for age-related macular degeneration. The study found that only 1 out of 35 treatment recommendations in the PPP cited a reliable intervention systematic review [8]. Our study complements the existing literature, highlighting that there may be areas to include additional supporting evidence in AAO PPPs.
In evidence-based medicine, randomized controlled trials (RCTs) and systematic reviews/meta-analyses synthesizing their results are the pinnacle of evidence as randomization reduces bias and allows for investigation of causal relationships. A study conducted in 2019 found that only 2% of all publications in the field of ophthalmology were RCTs [9]. In our study, while the proportion of LOE I (ie, meta-analysis, systematic reviews of RCTs, or RCTs) recommendations has increased from prior PPPs to current PPPs, this increase is primarily driven by a disproportionate underreporting of lower-level evidence. In fact, the number of LOE I recommendations has not increased. On the one hand, this trend suggests guideline authors may have attempted to highlight LOE I recommendations in the current PPPs. On the other hand, fully reporting both level I and lower-level evidence could help to expand the evidence base highlighted in ophthalmology guidelines.
The Institute of Medicine’s landmark reports on clinical practice guidelines were the impetus for the initial development of many guidelines in effect today [10]. In 2011, the Institute of Medicine recommended standards for developing trustworthy clinical practice guidelines [11]. The standards state that for each recommendation in a guideline, “a rating of the level of confidence in the evidence underpinning the recommendation” should be provided [11]. Our results suggest that substantial underreporting of LOE may exist in current PPPs, as the number of recommendations with reported LOE fell from 1254 in prior PPPs to 94 in current PPPs. These results suggest that there is significant opportunity to include level II and III evidence, which, despite risk of bias, is nonetheless often critically important data.
Use of data sources such as insurance claims or multi-institutional registries can provide information about real-world clinical practice that cannot be generated by randomized clinical trials. Furthermore, there are clinical questions for which a randomized trial is infeasible, such as for rare conditions or for procedures where shams are not possible, and in these circumstances, lower levels of evidence ought to be weighted more heavily. Since many ophthalmologic diseases have a low incidence and a heavy reliance on surgical management in certain subspecialties, performing randomized controlled trials may be especially challenging. By acknowledging this and including varied levels of evidence in ophthalmology PPPs, authors may be able to more easily adopt the Institute of Medicine’s recommendation about LOE reporting in clinical practice guidelines.
The same report from the Institute of Medicine also proposed that “recommendations should be articulated in a standardized form detailing precisely what the recommendation action is, and under what circumstances it should be performed” [11]. Our findings demonstrated that aside from indication of recommendation strengths (n = 104 recommendations across 24 PPPs), the current guidelines do not articulate recommendations in a standardized form.
Without such standardization, our comprehensive review of the 2021 Cataract in the Adult Eye PPP identified 510 statements that addressed how patients should be diagnosed or managed clinically and thus could be considered recommendations. While the agreement between our two reviewers was good (k statistic = 0.64), this result suggests that interpretations of potential recommendation statements in the PPP can be variable. Standardized articulation of recommendations would help clinicians clearly identify recommended actions for clinical practice. For example, American Heart Association guidelines list all recommendations in visually distinctive boxes, which stand out from the surrounding text and include levels of evidence [12]. Clearly articulating recommendations could also facilitate the creation and assessment of programs to improve the quality of care.
As health care usage and expenditure continue to rise in the United States, value-based care has become an increasingly important concept [13]. A recent systematic review found that between 75.7 and 101.2 billion was spent on low-value care in the United States [14]. Clinical practice guidelines play an important role in shaping practice patterns and thus may be well-suited to promote high-value care. In this area, prior work in cardiology has evaluated cost and value considerations in contemporary heart failure clinical guidelines [7]. The study concluded that although most contemporary heart failure guidelines contained cost/value statements, they were rarely used to support clinical guidance recommendations.
In the ophthalmology guidelines, a majority (88%) of PPPs included cost/value statements. In particular, the high economic impact of disease or care was frequently highlighted (75% of PPPs). However, cost/value considerations have yet to be incorporated into the development of specific recommendations, representing an avenue for future work in ophthalmology guideline development. More than half of the PPPs also reported gaps in cost/value evidence—ongoing efforts in the field such as the IRIS® Registry, which includes performance metrics, may facilitate real-world evidence generation in this area and help to provide needed data for guideline development [15].
The strengths of this study include analysis of all PPPs spanning a 10-year period, including all contemporary PPPs and their immediate predecessors. This thorough analysis allowed us to assess evolutions of PPPs over time and trends in all the specialties and topics that PPPs cover. Furthermore, we reported levels of evidence exactly as described in the guidelines. Additional strengths of this study included independent two-party grading, with validation by a board-certified anterior segment specialist, of levels of evidence for our review of the 2021 Cataract in the Adult Eye PPP.
The limitations include the potential underreporting of LOE in current PPPs, which prohibits us from drawing conclusions about all evidence supporting PPPs. This is partially addressed by our comprehensive review of the 2021 Cataract in the Adult Eye PPP, including its references, which suggests substantial underreporting of LOE across all LOE and disproportionate underreporting of lower-level evidence.
In conclusion, we performed a systematic analysis of reported LOE supporting AAO PPP guidelines. Compared with prior PPPs, current PPPs emphasize evidence from randomized controlled trials. While underreporting of LOE across all LOE exists, there appears to be a disproportionate underreporting of lower-level evidence. Future guideline development may consider clearly defining recommendations, explicitly reporting LOE associated with each recommendation, and integrating cost/value considerations in recommendations.