DOI: https://doi.org/10.21203/rs.3.rs-2397749/v1
Background
Clinical Practice Guidelines (CPGs) are standardized recommendations to improve healthcare and facilitate clinicians' decisions. We aimed to evaluate the quality of CPGs in paediatric dentistry using the AGREE II tool.
Methods
PubMed, EMBASE, Scopus, LIVIVO, Lilacs, international guidelines websites, scientific societies, and grey literature databases were searched until September 2021 by two reviewers. We included CPGs that contain paediatric dental recommendations, while drafts or guidelines designed for special needs patients were excluded. Descriptive statistics for the characteristics of the guidelines and mean overall domain scores (95% confidence interval) were calculated.
Results
Forty-three guidelines were included in this study. Overall mean (95% CI) for all the domains as follows: 1) scope and purpose (49.1%, 95%CI: 41.2–57), 2) stakeholder involvement (32.2%, 95%CI: 24.2–40.1), 3) rigour of development (29%, 95%CI: 22.3–35.7), 4) clarity of presentation (57.3%, 95%CI: 50.2–64.5), 5) applicability (15.8%, 95%CI: 10.4–21.3), and 6) editorial independence (37.7%, 95%CI: 26–49.5). Only one guideline was reported with scores ≥ 60% for all 6 domains.
Conclusions
The reporting quality of paediatric dentistry guidelines does not meet the standard for methodological quality, especially in domain 5 for applicability. Bias can be introduced during the development of clinical guidelines, which could mislead paediatric dentists and harm patients. Our results could help in establishing good quality CPGs with reliable recommendations by pointing out the importance of implementing the AGREE II tool with a system to evaluate the level of evidence.
Registration: The protocol of this study was prospectively registered on Open Science Framework - DOI (10.17605/OSF.IO/BFNGW).
Paediatric dentistry is an integrated field concerned with preventing, diagnosing, interception, and treating oral health problems in children and adolescents.1 These problems affect the growth and development process; by avoiding these risk factors and implementing early management programs to prevent and minimize oral health drawbacks allow the integral development from the maternal period through adolescence.2 Henceforth, clinicians' decisions should rely on evidence-based best practices in the best interest of the patient's health.3
The American Dental Association (ADA) defined Evidence-Based Dentistry (EBD) as an "approach to oral health care that requires the judicious integration of systematic assessments of clinically relevant scientific evidence, relating to the patient's oral and medical condition and history, with the dentist's clinical expertise and the patient's treatment needs and preferences".4 Therefore, best evidence should be originate from the highest quality and level of evidence (e.g., types of studies such as clinical trials, systematic reviews, among others).5 To comply with the EBD, Clinical Practice Guidelines (CPGs) have been developed, which are systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances. Hence, high-quality CPGs are considered the gold standard for translating evidence to clinical practice, achieving cost-effective integration of patient care with clear recommendations.6 Therefore, CPGs should be developed through rigorous methodologies that establish their quality. However, there are potential biases or conflicts of interest when conducting a CPG that could lead to inappropriate recommendations based on insufficient or poor evidence, leading to potential patient harm.7
Several paediatric dental societies and organizations have published CPGs focusing on multiple topics in paediatric dentistry; however, some of them might not be properly designed, and their recommendations are not evidence-based.8 Moreover, not all of them follow systems to evaluate the strength of evidence, such as the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach, a grading system designed to rate the certainty of a body of evidence in systematic reviews and CPGs.9
Due to the increase in published CPGs and recommendations, an international group from thirteen countries decided to develop policies to report on the quality of CPGs and ensure their high quality before endorsing them.10 The APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION II (AGREE II) instrument is a generic tool developed to ensure the quality of publishing guidelines and recommendations.10–12 The use of a reporting tool while conducting CPGs ensures transparency, facilitates evidence presentation, and links recommendations to the strength of evidence. A study8 evaluated the guidelines in paediatric dentistry restricted to CPGs published in the English Language up to November 2007 and evaluated the reporting quality using AGREE tool with its old version. Thus, we aimed to evaluate the quality of published CPGs and recommendations related to paediatric dentistry using AGREE II tool.
This is a meta-research based on the research question: "Do the clinical practice guidelines on paediatric dentistry meet the quality standards according to the AGREE II instrument?". The study protocol was registered prospectively on Open Science Framework - DOI (10.17605/OSF.IO/BFNGW).
We planned on our registered protocol, including best practice documents and used the AGREE REX tool to evaluate them. However, after reading the full text of the best practice document, we found that these documents were not developed based on a transparent methodology compared to the guidelines. After a joint discussion, our concern was that by including the evaluation of these documents, they would result in false estimation for the overall mean scores for the domains of evaluated CPGs through AGREE II tool. Therefore, we decided not to include the best practices in our study and not use the AGREE REX tool.
In addition, we did not calculate the intraclass correlation coefficient (ICC) for the scores between the reviewers. We considered that the consistency of what reviewers reported with their scores is more important than agreement. We allowed the variation between the reviewers up to 2 points, but if varied by ≥ 3 points, a consensus was reached after a discussion. We intended to perform linear regression analysis; however, as recommended by AGREE tool, we cannot aggregate the scores of 6 domains into a final total score. Moreover, we preferred to focus on and point out the flaws of reporting the quality of the guideline through AGREE II tool and how to improve it.
A literature search until September 2021 was conducted to identify the guidelines and recommendations related to paediatric dentistry in the following databases: Medline (PubMed), Excerpta Medica Database (EMBASE), Scopus, LIVIVO, and Caribbean Health Sciences Literature (LILACS). In addition, Scientific societies and International Associations' websites were searched, such as: the British Society of Paediatric Dentistry - BSPD, American Academy of Paediatric Dentistry - AAPD, European Academy of Paediatric Dentistry - EAPD, Guidelines International Network (GIN), National Institute for Health and Care Excellence – NICE, Turning Research Into Practice database - TRIP and Scottish Dental Clinical Effectiveness Programme - SDCEP. Furthermore, grey literature was searched through ProQuest to identify possibly eligible literature not identified during database searches. There were no restrictions on the language or the year of publication. However, if the Clinical Practice Guideline (CPG) has an update, the most recent one was considered.
The key terms used in the search strategy for this study were: (Paediatric Dentistry), (primary teeth and Permanent teeth), (Guideline), (clinical practice guideline), (Recommendations) with terms from a controlled vocabulary (MeSH terms), keywords, synonyms, related terms, combined with boolean operators “OR” and “AND”. See supplementary material 1 for the databases search strategy and websites of paediatric dentistry CPGs.
A- CPGs that contain a "statement" or "guideline" or provide "recommendation" directed to paediatric dentists;
B- Guidelines that included at least one recommendation related to paediatric dentistry based on literature and expert opinion.
A- Guidelines designed specifically for patients with special needs in paediatric dentistry;
B- Protocols for the development of CPGs and draft of the CPGs;
C- Outdated guidelines when the most updated is retrieved.
Two independent reviewers (RAE and CLG) were responsible for screening, eligibility, and inclusion of the studies. Any discrepancies were resolved by a joint discussion with a third expert reviewer (DPR). The identified references were uploaded to the EndNote reference management program web (https://www.myendnoteweb.com) to remove the duplicates after screening title/abstract/author/year, and the remaining results were exported to Rayyan13. We used Rayyan as it is a free web-tool (Beta), allow storage of the data for long time, and to facilitate for the reviewers working on phase 1 and 2. For the references that were collected in Excel form, were uploaded manually. For both methods, after duplicates were removed, documents were screened through phase 1, and documents were analysed based on the title and abstract. Documents that did not meet the inclusion criteria described above were eliminated. In phase 2, a full-text read was performed to check the exclusion criteria.
All the supplementary documents related to the CPGs were collected to be analysed. Two reviewers (RAE, CML) were responsible for extracting the following data from the included CPGs, independently: name of organization, the language, number of the author(s) and organizations, whether it is single or multicentre, country, publication year, title, name of the journal, journal's impact factor according to Journal Citation Report (JCR) 2020, the system used for level of evidence and grading of recommendations, study's purpose, target population, development group, target users, systematic methods of evidence search, evidence strengths and limitations, recommendations methods, evidence eligibility criteria, benefits/side effects /risks, expert peer-reviewed, guideline update procedure, recommendation-evidence linkage, concrete/unequivocal recommendations, problem/ disease management options, identifiable recommendations, application facilitators/barriers, how-to put recommendations into advice, application of costs, monitoring/auditing criteria, content influence by funding, development group competing interests.
The two reviewers (RAE, CML) completed an online training tool recommended by the AGREE developer on the website (www.agreetrust.org). Then we performed a pilot data extraction and evaluation on one CPG that was not included in our analysis. Afterwards, the two appraisers independently evaluated each guideline according to the domain-based instrument Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument user manual.10,12 The original AGREE instrument was developed in 2013 and then updated in 2017 as AGREE II, which consists of 23 items grouped under six domains. Each domain aims to assess each quality aspect of the guidelines, compromising: Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence. Furthermore, an overall assessment includes the rating of the overall quality of the guideline on a scale from 1–7, and whether we recommend the guideline for use, recommend with modification, or do not recommend it for use in practice.
Each item was rated on a scale from 1–7 (Likert scale), in which 1 refers that the concept is very poorly reported (strongly disagree) and 7 refers if the full criteria are reported (strongly agree). The included guidelines were evaluated by each reviewer independently through the "MY AGREE PLUS" function on AGREE II website. The scores were assigned based on each item's integrity report and full criteria. The 6 domains are independent and were not aggregated into a single quality score, as the overall assessment is an individual assessment and the AGREE II manual did not declare a method to judge the second question. Hoffmann-Eßer et al. (2017)14 found that domains 3, 4, and 5 had the most substantial influence during the conduction of CPGs. Therefore, we recommended a guideline for use when two of the three domains with a score ≥ 60%, recommended for use with modification when two domains were between 30–60%, and not recommended for use when two domains were ≤ 30%. If we found a guideline to receive equal distribution between the three domains, we decided according to the score of domain 3 as it had the strongest influence.
Inter-rater reliability and agreement was measured using Cohen's kappa test for eligibility criteria. The characteristics of the CPG were reported using descriptive statistics. In addition, descriptive statistics were calculated using each researcher's total score and the score per domain. To calculate the domain score, all item scores within the domain were aggregated and transformed into a maximum score of 100% for each item. Afterwards, each domain's mean overall scores and 95% confidence interval (CI) were calculated. For this purpose, we performed the statistical analysis using the Jamovi Software Version 1.2.27.0. (Sydney, Australia).
3.1 Guideline selection
The initial search identified 4,894 documents, 128 full-text articles were screened for eligibility, and 43 guidelines were included in our analysis (Figure 1). The inter-rater reliability and agreement using the Cohen kappa coefficient was k=0.67 (substantial agreement). Supplementary material 2 presents the title of included guidelines with the organization's name.
3.2 Guidelines demographics
Table 1 provides a general characteristic of the included CPGs. Seventeen (17) were developed in North America (13 in the United Sates of America and 4 in Canada), 25 in Europe (19 in the United Kingdom, 1 in Finland, 2 in Germany, 1 in Switzerland, 2 in Greece), and 1 in Asia (Malaysia). Around 39.5% (n=17) CPGs did not use any system to evaluate the level of evidence or grading of recommendations, while 25 CPGs used different systems, and one CPG did not declare the system's name. Only eight CPGs declared the use of AGREE tool, and 33 did not use any tool to develop the guidelines. The majority of guidelines (n=35) were single centre.
3.3 Quality assessment of guidelines and AGREE II scores
Table 2 presents the mean score (95% CI) and minimum and maximum values for all 6 domains of AGREE II tool for the included guidelines. The lowest mean score was for domain 5: Applicability (15.8%, 95% CI: 10.4–21.3). The highest mean score was for domain 4: Clarity of presentation (57.3%, 95% CI: 50.2–64.5). Only one guideline conducted by Scottish Dental Clinical Effectiveness Programme (SDCEP), "Conscious Sedation in Dentistry- third edition", was reported with a score >60% for all 6 domains. The quality evaluation of the 43 guidelines with the mean scores of 6 domains of AGREE II tool and the overall assessment are presented in supplementary material 3.
3.3.1 Scope and purpose
This domain is concerned with evaluating the guidelines' overall objectives, research question and target population. Only 34.9% (n=15) guidelines had scores ≥ 60% and 25.6% (n=11) had scores ≤ 30%.
3.3.2 Stakeholder involvement
Relates whether the guideline was developed by declared stakeholders, the involvement of relevant professional groups and whether the developers have considered the view and preferences of the target population. Seven guidelines were reported with scores ≥ 60%, while 53.5% (n=23) guidelines presented with scores ≤ 30%.
3.3.3 Rigour of development
This domain concerns methodological search, the process of formulating the recommendations and future update methods. Only 9.3% (n=4) of guidelines presented scores ≥ 60%, while the majority (n=26) of the guidelines reported scores ≤ 30%.
3.3.4 Clarity and presentation
This domain addresses the presentation and format of guidelines. Generally, most of the guidelines performed well; 19 presented scores ≥ 60%, while five reported scores ≤ 30%.
3.3.5 Applicability
This domain focuses on evaluating barriers, facilitators, and implementation of CPGs into practice. Only two guidelines were reported with scores ≥ 60%, while 34 guidelines were reported with scores ≤ 30%.
3.3.6 Editorial independence
This domain evaluates the funding declaration and conflict of interest. Nineteen guidelines did not report the funding source and conflict of interest declaration, which received a score of 0%. While five guidelines fully reported both funding source and conflict of interest statement and reported with a domain score of 100%.
3.3.7 Overall assessment
We recommend four guidelines for use, 14 for use with modification, while 25 are not recommended for use (Figure 2).
Our study is the most recent to evaluate CPGs in paediatric dentistry without publication year or language restrictions using the updated AGREE II instrument. We identified and included 43 guidelines addressing different topics in paediatric dentistry. Generally, the quality of the guidelines was poor, with flaws reported for domains 2, 3, and 5. Considering the clinical implications of the present study, we observed that the low quality of CPGs may lead to introduction of bias into the recommendations, might mislead readers into unreliable decisions, neglecting the importance of including the views and preference of the target population. These CPGs aimed to provide proper healthcare and improve patient outcomes; thus, it is essential to use a standardized appraisal instrument to conduct, update, and evaluate the clinical guidelines. Medical field studies have reported similar results to our study, such as: in CPGs for rehabilitation after anterior cruciate ligament reconstruction15 and in critical care CPGs.16 Additionally, in the dental field, quality assessments of CPGs have been carried out, such as paediatric dentistry8, orthodontics17, common dental procedures18, and dental clinical practices CPGs19, and found that the quality of dental guidelines is poor and inadequate about the AGREE instrument.
Contrariwise, one study evaluated the quality of traditional medicine CPGs20 reported a poor quality for "clarity of presentation", and the key recommendations were not easy to find. Most of our included guidelines and recommendations were well-described, and the key recommendations were well-organized. In our study, the only guideline that obtained a score of 100% for clarity of presentation was developed by SDCEP and other collaborators institutes (Prevention and management of dental caries in children "second edition"). Almost all guidelines failed to consider the "Applicability" without reporting the items. 93% (n=40) did not report the facilitators and potential barriers, 74.4% (n=32) did not present cost implications to guidelines, and 20.9% (n=9) were unclear in reporting the cost as they did not report the methods by which cost information was sought. This may lead to guidelines that recommend costly interventions or not applicable recommendations for certain healthcare situations. Furthermore, CPGs that have not reported any monitoring or audit criteria might not provide appropriate implementation to practice.
Following orthodontic CPGs17 for the domain "Rigour of Development", we observed a lack of declaring the systematic methods or the complete search strategy and eligibility criteria. 30.3% (n=13) have flaws in systematic methodology, eight guidelines did not have a systematic search, and seven guidelines conducted a systematic review and considered it the base for their recommendations which may result in potential bias and compromise recommendations' validity. Although some guidelines in our study declared using the AGREE II tool, we observed that they did not follow all the items properly. The potential bias arises as they should perform a methodological search and declare the complete search strategy and detailed analysis for the level of evidence through proper tools such as GRADE inside the guideline to ensure transparency.
Attention is needed for domain 6, "Editorial Independence," which focuses on the declaration of funding source, the role of funders and whether the development group members addressed the competing interests. Our results showed that the mean score is relatively low, and further improvement is needed. Several studies suggested increasing the awareness of this domain as it could be considered another source of bias.16,21,22 Of the 18 guidelines which declared the funding source, ten did not report the role of funders. Nineteen (19) CPGs fully reported the conflict of interest statement with the declaration on how the competing interests influenced the guideline process. Two CPGs declared that competing of interest was upon request. It is recommended to declare the funding and conflict of interest statement inside the guideline for the readers to ensure transparency and align to Open Science Practice concepts.
The AGREE Collaboration defined quality of guideline as the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid and feasible for practice.23 Furthermore, this tool has been endorsed by the Guideline Review Committee of the World Health Organization (WHO) and recommended for guideline developers and assessors.24 However, we noticed two topics for AGREE developers that should be improved. First, in their manual, there is no clear threshold to determine the quality of CPGs. Therefore, the quality of the guideline would be an individual appraiser's decision, and the scores must be interpreted carefully and considered in the appropriate context. Second, the AGREE II instrument evaluates the methodological process and guideline structure but does not evaluate the scientific evidence of recommendations. Therefore, some CPGs followed AGREE tool and received high scores in our study, but their recommendations might be biased, contain unreliable/low-quality informations, or have inappropriate inclusion/exclusion criteria. Although we did not recommend 25 guidelines for use according to AGREE tool assessment, they might include good clinical recommendations\ advice underpinned by reliable evidence. However, as the guidelines’ developer did not use a tool to conduct it, they resulted in poor reporting methodology.
In the present study, two reviewers appraised the included guidelines as recommended by AGREE II Collaboration12, considered a strength to increase the reliability of the assessment. We did not have any language restrictions and included two German language and 1 Finnish Language CPGs, the German Language CPGs were translated into English and contacted to the corresponding developer for the Finnish Language and provided us with English Language version. We aimed to include the eligible CPGs regardless to the language to ensure evaluating most of the available CPGs in pediatric dentistry. A limitation was observed while using AGREE tool that developers of the tool did not define how the appraisers should decide the overall assessment and did not consider the relative importance of a domain over the others (i.e., rigour of development is considered of equal importance to the other five domains).25 We overcame this drawback by searching the literature. We found a systematic review14 which examined the 6 domains to determine which domain strongly influences the overall assessment. The NICE organization has decided to close their evidence search on March 2022, focusing on delivering its priorities over the next five years. Since the mid of the COVID-19 pandemic, they have noticed the importance of living guidelines (i.e., up-to-date recommendations based on the latest evidence, practice, and technologies). This affected the reproducibility of our search strategy for NICE evidence search, but all the included CPGs retrieved from this website are accessible from bibliographic databases “the providers' websites”.
Awareness of future guideline updates should be considered to improve the methodological reporting of recommendations based on reliable evidence and involvement of the view and preference of the target population. Furthermore, great attention should be emphasized to reporting for domains "Applicability", "Rigour of Development", "Stakeholder Involvement", and "Editorial Independence". Correspondingly, we recommend implementing the AGREE II tool while conducting the best practices and policies documents, as they contain direct recommendations to clinicians without providing the formatting process or the strength of evidence. Paediatric dentists should be aware of this while implementing recommendations into practice. Guideline developers should use and adhere to AGREE II instrument while conducting and updating the guidelines to ensure transparency and improve applicability.
The reporting quality of paediatric dentistry guidelines does not meet the standard for methodological quality, especially in domain 5 for applicability. Our results can contribute to the improvement of future development for guidelines and influence the selection and use of guidelines in pediatric dental clinical practices.
(CPGs) Clinical Practice Guidelines
(ADA) American Dental Association
(EBD) Evidence-Based Dentistry
(GRADE) Grading of Recommendations Assessment, Development, and Evaluation
(AGREE II) The APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION II
(EMBASE) Excerpta Medica Database
(Lilacs) Caribbean Health Sciences Literature
(BSPD) the British Society of Paediatric Dentistry
(AAPD) American Academy of Pediatric Dentistry
(EAPD) European Academy of Pediatric Dentistry
(GIN) Guidelines International Network
(NICE) National Institute for Health and Care Excellence
(TRIP) Turning Research Into Practice database
(SDCEP) Scottish Dental Clinical Effectiveness Programme
(ICC) Intraclass Correlation Coefficient
(CI) Confidence Interval
Ethics approval and consent to participate
Not applicable
Consent for publication
Not applicable
Availability of data and materials
All data generated or analysed during this study are included in this published article [and its supplementary information files].
Competing interests
The authors declare that they have no competing interests
Funding
This study was supported by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) (Coordination for the Improvement of Higher Education Personnel)– finance code 001 granting scholarships to R.A.E and C.L.G. Conselho Nacional de Desenvolvimento Científico e Tecnológico (National Council for Scientific and Technological Development) (CNPq) provides scholarship awards for research productivity in Brazil to Dr. Raggio, Dr. Braga, Dr. Mendes. The funders had no role in the conduction of the study, interpretation of the data or decision to submit the manuscript for publication.
Authors' contributions
D.P.R, M.M.B, F.M.M, R.A.E, and C.L.G. conception and design of the study; R.A.E., C.M.L. acquisition of data; R.A.E., T.K.T., T.G. and D.P.R. analysis and interpretation of data; R.A.E. and C.M.L. drafted the manuscript; D.P.R., M.M.B., F.M.M., T.K.T., T.G., and C.L.G. revised and gave final approval of the manuscript.
Acknowledgements
We would like to thank Dr. Jonas de Almeida Rodrigues for his kind help in revising the translation of included German Language guidelines.
Table 1: Characteristics of included guidelines (n=43)
|
characteristics |
All guidelines- n (%) |
|
Year of publication
2000-2006 2007-2011 2012-2016 2017-2021 |
5 (11.6) 9 (21) 13 (30.2) 16 (37.2) |
|
Development organizations AAPD EAPD Finnish Medical Association AWMF, DGZ, DGZMK SDCEP WHO Ministry of Health Malaysia USPSTF BSPD UK national Guideline NICE ADA IADT |
10 (23.3) 7 (16.3) 1 (2.3) 2 (4.7) 4 (9.3) 1 (2.3) 1 (2.3) 1 (2.3) 5 (11.6) 4 (9.3) 1 (2.3) 2 (4.7) 4 (9.3) |
|
Number of centers[i] ≤2 ≥3 |
35 (81.4) 8 (18.6) |
|
Continent of published guidelines Asia Europe North America |
1 (2.3) 25 (58.1) 17 (39.5) |
|
Tools used to conduct guidelines No tool AGREE tool Other tools[ii] |
33 (76.7) 8 (18.6) 2 (4.7) |
|
Identification of “guideline” or “recommendation” in the title No Yes |
5 (11.6) 38 (88.4) |
|
Evidence grading system No system Unclear[iii] GRADE SIGN Other grade system[iv] |
17 (39.5) 1 (2.3) 12 (27.9) 11 (25.6) 2 (4.7) |
|
Language of published guideline English German Finnish |
40 (93) 2 (4.7) 1 (2.3) |
[i] When the developer’s organizations ≤2 we considered it single center, while number of organizations are ≥3, we considered a multicenter.
[ii] Other tools such as (special tool for the organization (developing NICE guidelines or WHO manual for development of guideline\ SIGN handbook).
[iii] Unclear as the guideline graded the recommendations and level of the evidence without declaring the system used.
[iv] Other systems such as (U.S. Preventive Services Task Force “USPSTF”).
Table 2: The mean score (95% CI), minimum and maximum values for all the 6 domains of AGREE II tool (n=43)
|
Domain |
Mean |
Min-Max % |
95%CI |
|
D1: Scope and purpose |
49.1 |
8–94 |
41.2–57 |
|
D2: Stakeholder involvement |
32.2 |
0–92 |
24.2–40.1 |
|
D3: Rigour of development |
29 |
2–83 |
22.3–35.7 |
|
D4: Clarity of presentation |
57.3 |
6–100 |
50.2–64.5 |
|
D5: Applicability |
15.8 |
0–63 |
10.4–21.3 |
|
D6: Editorial independence
|
37.7 |
0–100 |
26–49.5 |