Quality and Methodology of Clinical Practice Guidelines on Antiviral Pharmacotherapy for COVID-19 During the Early Phase of the Pandemic

Abstract

Background

Despite availability of reliable guidelines development methods, the risk of producing less reliable documents may be higher when the guidelines are developed rapidly and high-quality evidence is lacking. The aim of this study was to assess the ways in which such documents, dealing with antiviral therapy, were developed in the early stages of COVID-19 pandemic and assess if recommendations were supported by evidence.

Methods

We performed the search for guidelines or similar documents, published before the results of any randomized controlled trials of COVID-19 treatment were available, that considered antiviral therapies and contained a recommendations for clinicians. The quality of the guidelines was assessed using the AGREE II-Global Rating Scale Instrument and series of dichotomous criteria based on the domain 3 of the AGREE II tool. We analyzed variables associated with the presence of recommendations, including strong ones, for antiviral therapy for SARS-CoV-2.  

Results

The analysis included 40 publications, of which we classified 17 as clinical practice guidelines and 23 – as guidance or statements. The median of quality of documents assessed with the AGREE II-GRS tool (overall quality assessment on a scale ranging from 1-7) was 2.0 (IQR 1.5–2.5). Most documents did not fulfill the rigour of guideline development quality criteria. The AGREE II-GRS scores did not differ significantly across the type of the document, issuing institution and the mode of publication. Overall, 62.5% of documents provided recommendations for the use of antiviral medications despite apparent lack of sufficient evidence supporting such treatments. Documents that contained recommendations supporting antiviral drug use tended to be of lower quality than those without such recommendations. Of the included documents, 75% were not updated within the 2 months after the publication of the first randomized controlled trial on COVID-19 antiviral therapy. 

Conclusions

Most guidelines or guidance documents published during the early phase of the COVID-19 pandemic were of poor quality, contained recommendations for the use of antiviral therapy for SARS-CoV-2 infection despite only very low quality of evidence available, and were not updated on a regular basis. This information may be of value during this or future pandemics.

Contributions To The Literature

• We have found that the most of guidelines and guidance documents published during early phase of COVID-19 pandemic were of very poor quality.

• Many of these guidelines contained unjustified recommendations (often strong) for the use of various antiviral drugs for SARS-CoV-2 infection.

• These non-evidence-based documents were potentially misguiding treatment for COVID-19.

• This results indicate that even during the pandemics all guidelines and guidance documents should be developed using the validated methodology.

Background

Clinical practice guidelines (CPGs) and position statements are widely adopted ways to disseminate medical knowledge and influence clinical practice. Methods of reliable guidelines development are available.[1, 2] One of the broadly accepted methodology for ensuring the lowest risk of bias is the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach.[3–5, 6] However, the risk of producing less reliable documents persists [7–8], which may translate into less than optimal management of patients. Such risk may be higher when the guidelines are developed rapidly, as observed during coronavirus disease 2019 (COVID-19) pandemic. To investigate such possibility we decided:

1. To assess the methods used to formulate clinical CPGs and position statements containing recommendations on the pharmacological treatment of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, published during the early phase of the COVID-19 pandemic (before data from randomized controlled trials [RCTs] became available)

2. To identify recommendations for antiviral pharmacotherapy that were based on very low quality evidence available in the early phase of COVID-19 pandemics and to search for factors associated with the publication of such recommendations

3. To provide suggestions that would facilitate the development of guidelines in the future in cases when adequate quality evidence is lacking.

Methods

We conducted a retrospective analysis of the guidelines on COVID-19 management. Included documents were published before April 1, 2020; were written in English or had an English translation available; were published in journals or on the websites of scientific societies, regulatory bodies, or scientific institutions; considered antiviral therapies; and contained a recommendation (or recommendations) for clinicians. Those documents were defined for the purpose of this study as: 1) a statement with recommendation, advice, suggestions, or tips; and/or 2) a statement that it is not possible to provide any recommendation because of the lack of data considered adequate. Adaptations of existing guidelines were also included. Papers indexed as original articles or reviews were excluded.

We performed the search for relevant documents between May 15 and 22, 2020, using MEDLINE and Embase databases as well as Google Search. Search strategies are presented in Supplementary Material. An additional search was performed to detect if any update of the previously identified guidelines had been released within the 2 months after the publication of the first RCT on antiviral COVID-19 treatment.[9] Details on search strategy are presented in Additional File 1. Identified documents were screened based on the title and abstract (database search) or full-text documents (Google search). After duplicate removal, full-text documents were assessed for eligibility by two authors (FM and WL) independently (flowchart available in Additional file 2). Any discrepancies between evaluators during the study selection process were resolved by discussion.

The quality of the guidelines was assessed using the AGREE II-Global Rating Scale Instrument (AGREE II-GRS).[10] This scale has 4 core items (process of development, presentation style, completeness of reporting and clinical validity), each rated from 1 (lowest quality) to 7 (highest quality). Additionally, there is an overall assessments of the guideline quality, similarly ranging from 1 (lowest quality) to 7 (highest quality) and two additional items, based on users’ answer to questions whether they would recommend the guideline for use in practice and whether they would make use of a guideline of that quality in their own professional decisions (both rated from 1 = strongly disagree to 7 = strongly agree). Each document was assessed by two appraisers and the mean scores are reported.[10]

In addition, to characterise rigour of guideline development more precisely we assessed included documents using series of dichotomous criteria based on the domain 3 of the AGREE II tool itself and World Health Organization (WHO) standards [2, 11, 12]. Specifically, we verified if the assessed documents: used existing methodology of guidelines development; contained information on formation of the working group or searching for evidence; provided strength of recommendations and references to recommendations; included conflict of interest (COI) information; rated quality of evidence; updated document when new evidence was available (which we defined as within two month from publication of the first RCT (lopinavir/ritonavir efficacy in COVID-19 [9]) as well as sought opinion of external reviewers. We classified documents as containing strong recommendations (using phrases “is recommended”, “should be used”) or weak recommendations (suggestions, using phrases such as “may be used, consider the use” etc) or no recommendation for use of antiviral therapy (recommendation not to use antivirals outside clinical trials or a statement that evidence is lacking). Detailed information on data extracted is provided in Supplementary Information.

Two authors (FM and WL) extracted the data, and assessed the quality of CPGs (data extracted are listed in Additional File 3). Any disagreements were resolved by discussion.

Results

The final analysis included 40 publications, of which 17 were clearly labelled as CPGs. The flowchart for study selection is presented in the Additional File 2 and the list of included documents in the Additional File 4. The quality of most documents, as assessed with the AGREE II-GRS tool, was poor, except one document that scored maximum points (Surviving Sepsis Campaign Guidelines)[13] and two other documents that were of adequate quality (produced by the WHO[12] and the American Thoracic Society-led International Task Force[15]) (Fig. 1, detailed scores for all included documents are presented in the Additional File 5). The AGREE II-GRS scores did not differ significantly across most categories (i.e., the type of the document, an issuing institution, or the mode of publication) (Table 1). Most documents did not fulfill the rigour of guideline development quality criteria (Table 2, detailed assessment for all included documents presented in Additional File 6).

Table 1. The AGREE II-GRS score of the identified documents

Table 2. Proportion of documents (out of 40) that fulfilled the quality criteria for guideline development (based on AGREE II tool and World Health Organization standards [2, 11]

Figure 1. Distribution of the AGREE II Global Rating Scale Instrument (AGREE II-GRS) scores for the identified documents, where 1 denotes the lowest quality and 7 denotes the highest quality.

Overall, 62.5% of documents (n = 30) provided recommendations for the use of antivirals, of which 12.5% (n = 5) provided strong and 50% (n = 25) provided weak recommendations. There were no statistically significant differences in the proportion of documents that contained recommendations for the use of antivirals across the type of documents, mode of publishing (peer-reviewed journal vs website), or the type of an issuing institution. Documents that contained recommendations supporting antiviral drug use tended to be of lower quality (p = 0.113; Fig. 2) than those without such recommendations, and the presence of strong recommendation was associated with lowest quality. In the stepwise logistic regression analysis, no variables consistently associated with recommendations for the use of pharmacological treatment were identified. Of the included documents, 25% were updated within the 2 months after the publication of the first RCT on COVID-19 antiviral treatment [9].

Figure 2. AGREE II Global Rating Scale Instrument (AGREE II-GRS) total quality score by the type of a recommendation for the use of antiviral drugs

Discussion

We have found that most of documents providing information on COVID-19 treatment that were published within about 3 weeks after the WHO declared the state of pandemic [16] were of poor quality and were developed without the use of the widely accepted methods.

During the outbreak of the COVID-19 pandemic clinicians sought treatment options. Guidelines and similar documents were crucial source of information on management. Our results demonstrate that most of these documents published in the early phase of the pandemic were of low quality and thus potentially misguiding clinical practice. Moreover, the quality was poor irrespective of the institution that had published the document: in most instances, neither government organizations nor professional societies were able to ensure that the basic quality criteria were met. The same was found for peer-reviewed journals. It could be argued that poor quality of the evaluated documents was due to time pressure or resource constrain, yet accepting this one needs to point out that some organizations were providing guidelines of higher quality. In addition, some of the reviewed documents were developed before the countries involved were actually struck by the pandemic, so we would not consider time constrain to be explanation for the poor quality.

In the first months of the COVID-19 outbreak the only available data on the effectiveness of antiviral drugs for SARS-CoV-2 was derived from in vitro studies and – indirectly – from the Middle East respiratory syndrome and Severe Acute Respiratory Syndrome epidemics.[17–22] Based on these data, some drugs were claimed to be potentially effective.[23–25] However, RCTs were urgently needed to confirm their efficacy and safety in patients with COVID-19. In our opinion, no data justified their routine use and in March 2020 WHO stated: “Use of investigational anti-COVID-19 therapeutics should be done under ethically approved, randomized, controlled trials”.[14] One may easily venture that unjustified recommendations presented in many of analyzed documents contributed to different clinical behavior, with frequent application of unproven therapies (example: hydroxychloroquine) [26–29] Of note most prestigious international panels including WHO and Surviving Sepsis Campaign were ready not to issue recommendation and instead urged rapid research. [13, 14] strong.

Our analysis has several limitations. First, we decided to include a broad range of document types, including guidance documents and statements. However, when the analysis was limited to documents labelled by authors as CPGs, the results remained unchanged. We deliberately included the different types of documents to identify as many documents on which physicians base their decisions as possible. Another potential limitation of our study is the use of AGREE II-GRS tool to quantify the guideline quality, which – as a simplified version of the AGREE II tool – is less precise and less widely used.[30] However, in our opinion, the use of a more precise tool would not affect the main conclusion because the quality of most documents was very low, with the majority of them not even fulfilling the basic quality criteria.

We believe that even during the pandemic all guidelines and guidance documents should be developed using the validated methodology (the use of GRADE is highly recommended); and regularly updated, preferably immediately after new evidence becomes available. However, because our results suggest the opposite, physicians should be able to identify the credible guidelines. The recommendations for antiviral drug use in numerous official documents might not only have resulted in patients receiving unnecessary, or even harmful, treatment but might also have been one of the factors limiting recruitment to RCTs in the first month of the pandemic.

Conclusions

Our findings indicate that in the initial stages of the pandemic practice advice / recommendations were of generally poor quality while including recommendations (frequently strong) for antiviral therapy. This observation should add those advocating new therapies in this, and possibly new clinical situations, and those following such advice in their own practice.

Abbreviations

AGREE II-GRS – AGREE II-Global Rating Scale Instrument; COI – conflict of interest; COVID-19 – coronavirus disease 2019; CPGs – Clinical practice guidelines; GRADE – Grading of Recommendations, Assessment, Development, and Evaluation; RCT(s) – randomized controlled trial(s); SARS-CoV-2 – severe acute respiratory syndrome coronavirus 2; WHO – World Health Organisation

Declarations

Ethics approval and consent to participate: Not applicable

Consent for publication: Not applicable

Availability of data and materials: The dataset created and analysed during the current study is available from the corresponding author on reasonable request.

Competing interests: None declared

Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Authors' contributions: FM and WL are responsible for study design, search, data extraction and analysis and writing the primary draft. RJ participated in planning the data analysis and writing the final version of the manuscript.

Acknowledgements: None

References

1. Appraisal of Guidelines, Research, and Evaluation in Europe (AGREE) Collaborative Group. Guideline development in Europe. An international comparison. Int J Technol Assess Health Care. 2000;16:1039–49.
2. Https://Instrument-2009-Update-2017.pdf. Accessed 16 May 2020.
3. Atkins D, Best D, Briss PA, et al. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:1490.
4. Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94.
5. Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64:401–6.
6. Https://. Accessed 16 May 2020.
7. Alexander PE, Brito JP, Neumann I, Gionfriddo MR, Bero L, Djulbegovic B, Stoltzfus R, Montori VM, Norris SL, Schünemann HJ, Guyatt GH. World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. J Clin Epidemiol. 2016;72:98–106.
8. Li Q, Li X, Wang J, Liu H, et al. Diagnosis and treatment for hyperuricemia and gout: a systematic review of clinical practice guidelines and consensus statements. BMJ Open. 2019;9:e026677.
9. Cao B, Wang Y, Wen D, et al. A trial of lopinavir-ritonavir in adults hospitalized with severe Covid-19. N Engl J Med. 2020;382:1787–99.
10. Https://. Accessed 16 May 2020.
11. Schünemann HJ, Fretheim A, Oxman AD, WHO Advisory Committee on Health Research. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006;4:3.
12. Https://. Accessed 16 November 2020.
13. Alhazzani W, Møller MH, Arabi YM, et al. Surviving Sepsis Campaign: guidelines on the management of critically ill adults with Coronavirus Disease 2019 (COVID-19). Intensive Care Med. 2020;46:854–87.
14. World Health Organization. (‎2020)‎. Clinical management of severe acute respiratory infection (‎‎‎SARI)‎‎‎ when COVID-19 disease is suspected: interim guidance, 13 March 2020. World Health Organization. https://apps.who.int/iris/handle/10665/331446. Accessed 16 May Accessed 16 May 2020.
15. Kevin C, Wilson SH, Chotirmall C, Bai. Jordi Rello on behalf of the International Task Force on COVID-19 COVID‐19: Interim Guidance on Management Pending Empirical Evidence. From an American Thoracic Society‐led International Task Force https://. Accessed 16 May Accessed 16 May 2020.
16. Https://. Accessed 16 May Accessed 16 May 2020.
17. Liu J, Cao R, Xu M, Wang X, et al. Hydroxychloroquine, a less toxic derivative of chloroquine, is effective in inhibiting SARS-CoV-2 infection in vitro. Cell Discov. 2020;6:16.
18. Anastasiou IA, Eleftheriadou I, Tentolouris A, Tsilingiris D, Tentolouris N. In vitro data of current therapies for SARS-CoV-2. Curr Med Chem. 2020;27:4542–8.
19. Wang M, Cao R, Zhang L, et al. Remdesivir and chloroquine effectively inhibit the recently emerged novel coronavirus (2019-nCoV) in vitro. Cell Res. 2020;30:269–71.
20. Sheahan TP, Sims AC, Leist SR, et al. Comparative therapeutic efficacy of remdesivir and combination lopinavir, ritonavir, and interferon beta against MERS-CoV. Nat Commun. 2020;11:222.
21. Arabi YM, Shalhoub S, Omari AA, et al. Effect of ribavirin and interferon on the outcome of critically ill patients with MERS. Am J Respir Crit Care Med. 2020;195:A6067.
22. Arabi YM, Mandourah Y, Al-Hameed F, et al. Corticosteroid therapy for critically ill patients with Middle East respiratory syndrome. Am J Respir Crit Care Med. 2018;197:757–67.
23. Martinez MA. Compounds with therapeutic potential against novel respiratory 2019 coronavirus. Antimicrob Agents Chemother. 2020;64:e00399-20.
24. Xu J, Shi PY, Li H, Zhou J. Broad Spectrum Antiviral Agent Niclosamide and its therapeutic potential. ACS Infect Dis. 2020;6:909–15.
25. Gul MH, Htun ZM, Shaukat N, Imran M, Khan A. Potential specific therapies in COVID-19. Ther Adv Respir Dis. 2020;14:1753466620926853.
26. Bull-Otterson L, Gray EB, Budnitz DS, et al. Hydroxychloroquine and chloroquine prescribing patterns by provider specialty following initial reports of potential benefit for COVID-19 treatment - United States, January-June 2020. MMWR Morb Mortal Wkly Rep. 2020;69:1210–5.
27. Bhatnagar T, Murhekar MV, Soneja M, Gupta N, Giri S, Wig N, Gangakhedkar R. Lopinavir/ritonavir combination therapy amongst symptomatic coronavirus disease 2019 patients in India: Protocol for restricted public health emergency use. Indian J Med Res. 2020;151:184–9.
28. Costanzo M, De Giglio MAR, Roviello GN. SARS-CoV-2: Recent reports on antiviral therapies based on lopinavir/ritonavir, darunavir/umifenovir, hydroxychloroquine, remdesivir, favipiravir and other drugs for the treatment of the new coronavirus. Curr Med Chem. 2020;27:4536–41.
29. Kalil AC. Treating COVID-19-off-label drug use, compassionate use, and randomized clinical trials during pandemics. JAMA. 2020;323:1897–8.
30. Seto K, Matsumoto K, Fujita S, Kitazawa T, Amin R, Hatakeyama Y, Hasegawa T. Quality assessment of clinical practice guidelines using the AGREE instrument in Japan: A time trend analysis. PLoS One. 2019;14:e0216346.

Tables

Table 1. The AGREE II-GRS score of the identified documents

*mean from the scores of 2 evaluators; (from 1 - lowest to 7 - highest)

Table 2. Proportion of documents (out of 40) that fulfilled the quality criteria for guideline development (based on AGREE II tool and World Health Organization standards [2, 11]