This is a meta-research based on the research question: "Do the clinical practice guidelines on paediatric dentistry meet the quality standards according to the AGREE II instrument?". The study protocol was registered prospectively on Open Science Framework - DOI (10.17605/OSF.IO/BFNGW).
2.1 Deviation from the Protocol
We planned on our registered protocol, including best practice documents and used the AGREE REX tool to evaluate them. However, after reading the full text of the best practice document, we found that these documents were not developed based on a transparent methodology compared to the guidelines. After a joint discussion, our concern was that by including the evaluation of these documents, they would result in false estimation for the overall mean scores for the domains of evaluated CPGs through AGREE II tool. Therefore, we decided not to include the best practices in our study and not use the AGREE REX tool.
In addition, we did not calculate the intraclass correlation coefficient (ICC) for the scores between the reviewers. We considered that the consistency of what reviewers reported with their scores is more important than agreement. We allowed the variation between the reviewers up to 2 points, but if varied by ≥ 3 points, a consensus was reached after a discussion. We intended to perform linear regression analysis; however, as recommended by AGREE tool, we cannot aggregate the scores of 6 domains into a final total score. Moreover, we preferred to focus on and point out the flaws of reporting the quality of the guideline through AGREE II tool and how to improve it.
2.2 Selection of the Guidelines and Recommendation
A literature search until September 2021 was conducted to identify the guidelines and recommendations related to paediatric dentistry in the following databases: Medline (PubMed), Excerpta Medica Database (EMBASE), Scopus, LIVIVO, and Caribbean Health Sciences Literature (LILACS). In addition, Scientific societies and International Associations' websites were searched, such as: the British Society of Paediatric Dentistry - BSPD, American Academy of Paediatric Dentistry - AAPD, European Academy of Paediatric Dentistry - EAPD, Guidelines International Network (GIN), National Institute for Health and Care Excellence – NICE, Turning Research Into Practice database - TRIP and Scottish Dental Clinical Effectiveness Programme - SDCEP. Furthermore, grey literature was searched through ProQuest to identify possibly eligible literature not identified during database searches. There were no restrictions on the language or the year of publication. However, if the Clinical Practice Guideline (CPG) has an update, the most recent one was considered.
The key terms used in the search strategy for this study were: (Paediatric Dentistry), (primary teeth and Permanent teeth), (Guideline), (clinical practice guideline), (Recommendations) with terms from a controlled vocabulary (MeSH terms), keywords, synonyms, related terms, combined with boolean operators “OR” and “AND”. See supplementary material 1 for the databases search strategy and websites of paediatric dentistry CPGs.
2.3 Eligibility criteria
2.2.1 Inclusion criteria
A- CPGs that contain a "statement" or "guideline" or provide "recommendation" directed to paediatric dentists;
B- Guidelines that included at least one recommendation related to paediatric dentistry based on literature and expert opinion.
2.2.2 Exclusion criteria
A- Guidelines designed specifically for patients with special needs in paediatric dentistry;
B- Protocols for the development of CPGs and draft of the CPGs;
C- Outdated guidelines when the most updated is retrieved.
Two independent reviewers (RAE and CLG) were responsible for screening, eligibility, and inclusion of the studies. Any discrepancies were resolved by a joint discussion with a third expert reviewer (DPR). The identified references were uploaded to the EndNote reference management program web (https://www.myendnoteweb.com) to remove the duplicates after screening title/abstract/author/year, and the remaining results were exported to Rayyan13. We used Rayyan as it is a free web-tool (Beta), allow storage of the data for long time, and to facilitate for the reviewers working on phase 1 and 2. For the references that were collected in Excel form, were uploaded manually. For both methods, after duplicates were removed, documents were screened through phase 1, and documents were analysed based on the title and abstract. Documents that did not meet the inclusion criteria described above were eliminated. In phase 2, a full-text read was performed to check the exclusion criteria.
2.4 Data extraction and Information handling
All the supplementary documents related to the CPGs were collected to be analysed. Two reviewers (RAE, CML) were responsible for extracting the following data from the included CPGs, independently: name of organization, the language, number of the author(s) and organizations, whether it is single or multicentre, country, publication year, title, name of the journal, journal's impact factor according to Journal Citation Report (JCR) 2020, the system used for level of evidence and grading of recommendations, study's purpose, target population, development group, target users, systematic methods of evidence search, evidence strengths and limitations, recommendations methods, evidence eligibility criteria, benefits/side effects /risks, expert peer-reviewed, guideline update procedure, recommendation-evidence linkage, concrete/unequivocal recommendations, problem/ disease management options, identifiable recommendations, application facilitators/barriers, how-to put recommendations into advice, application of costs, monitoring/auditing criteria, content influence by funding, development group competing interests.
2.5 Quality of evidence assessment and Appraisal with AGREE II tool
The two reviewers (RAE, CML) completed an online training tool recommended by the AGREE developer on the website (www.agreetrust.org). Then we performed a pilot data extraction and evaluation on one CPG that was not included in our analysis. Afterwards, the two appraisers independently evaluated each guideline according to the domain-based instrument Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument user manual.10,12 The original AGREE instrument was developed in 2013 and then updated in 2017 as AGREE II, which consists of 23 items grouped under six domains. Each domain aims to assess each quality aspect of the guidelines, compromising: Scope and Purpose, Stakeholder Involvement, Rigor of Development, Clarity of Presentation, Applicability, and Editorial Independence. Furthermore, an overall assessment includes the rating of the overall quality of the guideline on a scale from 1–7, and whether we recommend the guideline for use, recommend with modification, or do not recommend it for use in practice.
Each item was rated on a scale from 1–7 (Likert scale), in which 1 refers that the concept is very poorly reported (strongly disagree) and 7 refers if the full criteria are reported (strongly agree). The included guidelines were evaluated by each reviewer independently through the "MY AGREE PLUS" function on AGREE II website. The scores were assigned based on each item's integrity report and full criteria. The 6 domains are independent and were not aggregated into a single quality score, as the overall assessment is an individual assessment and the AGREE II manual did not declare a method to judge the second question. Hoffmann-Eßer et al. (2017)14 found that domains 3, 4, and 5 had the most substantial influence during the conduction of CPGs. Therefore, we recommended a guideline for use when two of the three domains with a score ≥ 60%, recommended for use with modification when two domains were between 30–60%, and not recommended for use when two domains were ≤ 30%. If we found a guideline to receive equal distribution between the three domains, we decided according to the score of domain 3 as it had the strongest influence.
2.6 Statistical Analysis
Inter-rater reliability and agreement was measured using Cohen's kappa test for eligibility criteria. The characteristics of the CPG were reported using descriptive statistics. In addition, descriptive statistics were calculated using each researcher's total score and the score per domain. To calculate the domain score, all item scores within the domain were aggregated and transformed into a maximum score of 100% for each item. Afterwards, each domain's mean overall scores and 95% confidence interval (CI) were calculated. For this purpose, we performed the statistical analysis using the Jamovi Software Version 1.2.27.0. (Sydney, Australia).