A Delphi consensus to advance on a Clinical Case De�nition for Post COVID-19 condition: A WHO protocol

Background: While most people developing COVID-19 recover quickly and completely, a proportion of those infected with SARS-CoV-2 develop persistent symptoms, which can be disabling. After establishing the interim nomenclature for Post COVID-19 condition, there is a need to establish a clinical case de�nition to be used by all parties in all settings. Of the many de�nitions available to date, none is universal and the WHO has engaged in formally de�ning the Post COVID-19 condition based on existing evidence and by means of the most robust methodology. A key element for success is to engage patients and advocacy groups in this exercise. The objective of the study “Delphi consensus to advance on a Clinical Case De�nition for Post COVID-19 condition” is to determine the domains, variables and values (thresholds) for inclusion into a clinical case de�nition for Post COVID-19 condition, using a Delphi consensus methodology. Methods: This study is a prospective, Delphi consensus-seeking exercise, mixed, iterative survey of internal and external experts, patients and other stakeholders. Panelists (individual participants who will �ll the online survey in) will ful�l the criteria of balanced representativity of all WHO regions, age and gender balance, specialty diversity, and patients. We


Introduction
As of April 22, 2021, over 143 million con rmed cases of coronavirus disease 2019 (COVID-19) and over 3 million deaths have been reported to WHO. [1] The natural history, clinical course and consequences of this new disease are not completely understood.[2] Most COVID-19 patients return to normal after the acute infection with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).But a proportion of survivors report ongoing health issues, such as fatigue, temperature, respiratory and mental symptoms, and new or worsening conditions, among others.The estimate of total persons affected with Post COVID-19 condition remains unknown, but published reports identify that about 10-20% of COVID-19 patients experience lingering symptoms for weeks to months following acute SARS-CoV-2 infection.
There is a need to integrate new ndings on the Post COVID-19 condition and carefully interpret the growing quantity of research from case series, clinical cohorts, surveys and registries led by public health agencies, academic universities, patient-led research groups and other stakeholders.One of the rst hurdles was to establish the interim nomenclature for the Post COVID-19 condition, as the list of related terminologies is long.Since January 2020 in response to Member States requests, the Classi cation and Terminologies unit has been progressively activating emergency codes for COVID-19 in ICD-10 and ICD-11 after consultation with relevant committees and reference groups of the WHO Family of International Classi cations (WHO-FIC) https://www.who.int/classications.WHO suggests the wider use of the term Post COVID-19 condition, as it is neutral and lacks attribution of causality, and already has designated ICD-10 code (U09: Post-COVID Condition) and ICD-11 codes (RA02: Post-COVID Condition) (Figure 1).[3] A second hurdle, is establishing a clinical case de nition of the Post COVID-19 condition, to be used by all parties in all settings.A list of related de nitions continues to grow (see Appendix 1).However, to date, none is universal and the WHO has engaged in formally de ning the Post COVID-19 condition in a way based on existing evidence.

Objective
The objective of this study is to determine the domains, variables and values (thresholds) for inclusion into a clinical case de nition for the Post COVID-19 condition, using a Delphi consensus methodology.

Study Design
This study is a prospective, Delphi consensus-seeking exercise, mixed, iterative survey of internal and external experts, patients and other stakeholders.A Delphi consensus methodology will be used to survey a panel of experts who as individual participants will ll the online survey in (from now on panelists).Panelists ful l the criteria of balanced representativity (age, sex, location, patients/relatives of, medical area of expertise Primary/Specialized, translational research, media/journalists, Health policy/payers, …)

Study Participants
We aim to sample approximately 100 participants including the Chairs, speakers and Working Group panelists of the 1 st WHO webinar entitled "Expanding our understanding of Post COVID-19 condition", held on February 9 th , 2021.This webinar was attended by 42 participants and was reasonably balanced by age (no formal stats), sex (42.9% female), and location.Importantly, it also included patients.
The primary users of a clinical case de nition for the Post COVID-19 condition will include: patients, relatives and caretakers, clinicians, researchers, advocacy groups, policy-makers, health and disability insurance, and media.We will therefore aim to have a diverse sample of healthcare workers with expertise in clinical care, quality improvement and/or research.Eventually, the Guideline Development Group and other WHO panels might be invited to participate, as well as individual experts and patient representatives, with an emphasis to enrich female, young and/or those in LMIC panelists.Some groups will be expanded considering broad representativeness.There are no speci c exclusion criteria for panelists.

Tools
The Delphi method is a structured communication technique originally developed as a systematic, interactive forecasting method which relies on a panel of experts.[4,5].Essentially, a Delphi process is one in which a panel of experts answers a series of questionnaires over two or more rounds in an attempt to achieve consensus.The technique can also be adapted for use in face-to-face meetings (termed "mini-Delphi" or Estimate-Talk-Estimate (ETE)).Delphi has been widely used for research and has certain advantages over another structured forecasting approaches.[6] Delphi is based on the principle that forecasts (or decisions) from a structured group of individuals are more accurate than those from unstructured groups.The panelists answer questionnaires in two or more rounds.After each round, a facilitator or change agent provides an anonymized summary of the panelists' forecasts from the previous round as well as the reasons they provided for their judgments.Thus, experts are encouraged to revise their earlier answers in light of the replies of other members of their panel.It is believed that during this process the range of answers will decrease and the group will converge towards the "best" answer.Finally, the process is stopped after a prede ned stop criterion (e.g., number of rounds, achievement of consensus, stability of results), and the mean or median scores of the nal rounds determine the results.[7] Special attention is paid to the formulation of the Delphi theses and the de nition and selection of the panelists in order to avoid methodological weaknesses that could threaten the validity and reliability of the results.[8] For this Delphi task, the COMET software will bus used.COMET COMET Initiative DelphiManager is a web-based system designed to facilitate the building and management of Delphi surveys.[9] From a simple Excel template, COMET creates a multi-round Delphi survey using the Setup Wizard.Panelists will rate each question using a 1-9 Likert scale with an "Unable to rate" option, should they feel that item is beyond their expertise.In order to avoid a participant selecting the "Unable to rate" column accidentally, it is colored differently by default.An example of what a participant might be shown in round 2 (of a Delphi with a single panel throughout) is given in Figure 2.
This survey can then be managed in terms of emailing participants regarding missing data, calculating the feedback data for individual rounds and downloading all user and scoring data in standard CSV format.

Study Procedures
Lists held by the Principal Investigator, and the research team members, will be used to identify eligible participants.Eligible participants will be invited to participate with a recruitment letter on-line to solicit participation and engagement, along with an explanation of the study objectives, instructions and outputs.Engagement and commitment will be sought prior to the rst Round of the Delphi.The survey will then be conducted electronically with a link provided to engaged participants.Returned survey responses will be on a server at the University of Liverpool (Prof Paula Williamson, personal communication).The survey will contain listed options regarding domains/variables to consider in the de nition, initially as broad and comprehensive as possible.The agreed domains/variables will be followed by a series of questions relating to these variables with eventual values/thresholds related to each.There will be no questions with participant identi ers or related to personal health information.Surveys will be anonymous.
At the DelphiManager Registration Page, all participants must register to take part in the survey from round 1.It also contains a consent checkbox in order to comply with GDPR legislation.All of these elds are mandatory by default.The designated administrator will allow panelists to log in and type their responses.Once the survey has been launched, the administrator will overview the main study results during the round as well as manage the process of moving from one round to the next using the Next Round Wizard.Information will include: Total participants registered; Total completed each round; Total not completed each round; A breakdown of the completed/not completed participants for each stakeholder group (if selected); and Total of any additional outcomes added to each round.All questions will be evaluated on a 9-point scale, from one (1) to nine ( 9) and participants will be asked to choose the level of importance for each variable in the de nition.Some reclassifying by label in categories 1-3, 4-6, 7-9 might be needed.
The rst round of the Delphi will last 5 days, and participants will be sent 2 reminders to complete the online survey.The survey is iterative as the Delphi methodology requires multiple rounds of questioning until consensus is achieved.During the second round, the same survey will be used.Respondents will be informed of question results from the rst-round questions as a single panel.The questions for which consensus is achieved will be removed from further rounds of the survey.In questions where consensus is not achieved participants will be asked to reconsider their answers for all questions in lieu of the panel's anonymized results.Panel answers will be provided as numbers of respondents having chosen that answer.A maximum of three rounds of questions will be performed.

Working proposal with suggested methodology based on recent experience
For this exercise, the following steps will be considered: 1. Simple survey, with wide (global, full) respondent base to rating all domains (variables) that would be important to be considered for post-COVID diagnostic criteria.Would likely just con rm existing list, but important to consult widely.
2. Three (each in a different time zone) online consensus workshops x ~25 participants each (total n=75 to 100) to discuss and establish threshold criteria (likely to be more di cult to get agreement than the domains themselves).Participants would be invited, representative of key stakeholder groups and have knowledge of the condition.May want to ask a few (selected) people to give *very short* presentations of relevant evidence.
3. If any remaining unresolved aspects of the de nition after the virtual workshops, run a (short 2round) Delphi (via the COMET software) to seek nal resolution.
4. (Optional) Put nal agreed case de nition out for a structured, short-response time public consultation.

Statistical Plan Primary and Secondary Endpoints
The primary outcome will be a consensus on the importance of variable inclusion into the registry.'Consensus' will be obtained on a question if 70% or more of the responses fall within the same response on a 9-point Likert scale.'Disagreement' will occur if 35% or more of responses fall in both of the two extreme ranges of possible options on the Likert scale.All other combinations of panel answers will be considered 'partial agreement'.For each question, consensus proportions will be considered based on the number of respondents (excluding the category "Not my area of expertise").Therefore, the denominator for the consensus will include only participants with knowledge and expertise for that speci c question.During each round, participants will have the opportunity to add comments for each item and add variables that are not mentioned (Round 1 only).

Statistical Methods/Data Analysis
All downloaded data from DelphiManager is presented in standard CSV format, allowing it to be easily imported into most statistical analysis tools.DelphiManager does produce the feedback in rounds 2 and 3 for particular study designs, but the data extraction facility allows the study team to undertake additional analysis as required.
Where there is a value in the DelphiManager Rating column that is something other than the Likert scale values of 1 to 9, the system is coded as follows: "-9": the value allocated when an outcome has not been rated; "10": the value allocated to the "Unable to rate" option.
Response rate in proportion will be calculated based on the number of returned surveys.The survey results will be used descriptively.Participant responses, including baseline and demographics characteristics (survey section I) will be analyzed using basic statistics such as mean (standard deviation), median (interquartile range), and range.Responses on all other domains will be analyzed in proportions and illustrated using histograms.Reliability will be measured using test-retest in 5-10 respondents, and measures of internal consistency will be analyzed using Cronbach's alpha, to assess how well items of the same construct are related to one another.
Registration of panelists and the actual Delphi questionnaire are accessible via this link https://delphimanager.liv.ac.uk/De ningPostCOVID/Delphi.

Sample Size and Power
The objective will be to obtain a diverse sample of panelists from all stakeholders, including each of the following categories: clinical experts, research experts, and patient representatives.Understanding that some panelists may be experts in more than one category, we will invite at least 20 experts from each of the three categories.This should lead to a sample size of approximately 75-100 participants and solidify the generalizability of the consensus ndings.Allowing for a 10-15% non-response (or non-desire to participate) rate, and 10% drop-out rate, we will invite and engage at least 100 participants (TBD).

Consent
Consent to participant in the survey will be implied by answering and returning the survey.A statement explaining implied consent is found on the title page of the COMET survey (see survey Title page).No explicit consent will be obtained.

Con dentiality of Data
Results of the anonymous questions will be entered into a database document on a secured and locked computer.Only the study team will have access to the stored survey data.
This study is funded internally in WHO.There will be no payments to panelists.Cost of the COMET DelphiManager licenses (however many rounds, participants, items, timescale) is £850 plus VAT, a second Delphi study would be charged at £500 plus VAT, a third and every one thereafter at £300 plus VAT; however, as this enquiry is related to the WHO's work on COVID-19, software would be made free of charge.(Prof Paula Williamson, personal communication).There are no other expected expenses.

A list of domains (and values)
An initial list of fourteen (14) domains was considered in a de nition of post COVID-19 condition.These domains were later reduced/revamped from 14 to only 10, to be asked to panelists in an online text questionnaire, available in Appendix 2. Before the actual questions there is a page with instructions on how to ll in responses and the score method, and on the full Delphi procedures.All to be uploaded to DelphiManager.

DelphiManager checklist
The DelphiManager checklist is ful lled, completed as a necessary step before starting the study -run (piloting) of DelphiManager software 21 April Questionnaire completion 22 April Second dry-run (piloting) of DelphiManager software 23 April Final investigators decisions 23 April Submission to PRC 23 April Posting in online repository: pre-print Protocol Exchange from Nature 26 April Start of individual mailing 3 May Fieldwork Round 1 (3-7 May) 10 May Fieldwork Round 2 (10-14 May) 17 May Panel discussion and recommendations # June 2021 Completion of the report/presentation at the 2 nd webinar in the series; Publication Anticipated Results is expected to obtain a clinical case de nition for Post COVID-19 condition by consensus of a representative group of panelists including patients and patient advocates, clinicians, researchers and other stakeholders.

Figures
Figures

Figure 1 Emergency
Figure 1