Evaluating COVID-19 Vaccination Website using DISCERN and QUality Evaluation Scoring (QUEST) Tools

Background: The objective of this paper was to evaluate the COVID-19 vaccine registration website across UN-recognized member states for their portal quality, reliability, Ease Use and help to the general population in informed decision making. Methods: 12 UN member states (Countries) were selected based on the inclusion and exclusion criteria mentioned in the methodology section of this paper. PPS technique was used for sampling and selection of 12 countries from 193 UN member states. Post selection of UN member states the study used 2 step evaluation techniques, Step 1 The DISCERN checklist consists of 16 questions in three sections and is aimed to assess the reliability of information and quality of information; Step 2 The QUEST tool consists of 6 items with a subitem, Authorship, Attribution, Conict of Interest, Complementarity, Currency, Tone. The authors developed a standard set of instructions for evaluating Vaccination Portals to bring uniformity in understanding and context setting. Results: DISCERN tool overall reliability score on the Likert scale of 0 to 5 was 4 (SD ± 1.28). On Quality of information regarding treatment choices, the average score was 3.4 (SD ± 1.67). The QUEST tool on ease of use, concision, and comprehensiveness demonstrated an average score of 18.1 (SD ± 8.3) out of 28. The vaccine registration portal of the Czech Republic was found to be most informative and was able to provide a piece of scientically valid information on safety, ecacy, long-term short effects, choice of vaccine with attributable authors details. India, Bangladesh, Nigeria, South Africa scored relatively low to missing critical information on the website. The United Arab Emirates, Republic of South Korea, Indonesia, Australia, and Argentina had minor elements missing. Conclusions: COVID-19 vaccination portals vary in the quality of information and many were found unable to provide critical information for decision making on getting vaccinated


Implications for policy-makers
The Study highlights the pressing need for introducing consumer health informatics approach while designing COVID-19 vaccination portals, the multisectoral involvement is a key to improve the consumer engagement and create conducive environment of mutual trust. The Vaccine hesitancy needs to be tackled in systematic way, and developing a valid, quality and reliable registration portal for COVID-19 vaccination should be considered as rst step by policy makers.

Implications for public
The COVID-19 pandemic has been pandemic of misinformation, and the vaccination campaign has been surrounded by speculations, rumours, this has lead to vaccine hesitancy amongst many. In such situation the Vaccine Registration Portal is the rst interface, and should contain valid, reliable information which would enable laymen to undertake and informed decision on vaccination. Our study focus was to evaluate the quality of Vaccine Registration Portals to seek improvement and improve vaccination coverage. Background COVID-19 vaccination has been a key public health issue of 2021 with countries globally investing heavily on providing vaccination coverage to their residents, with over 1.62 billion individuals globally received at least one dose of vaccine and over 383 million individuals globally fully vaccinated (4.9% of the global population) as on 23rd May 2021 1 . The investments and research channelled towards the development and distribution of vaccine are evident with 26 vaccine candidates being available for human use and billions of dollars invested for the development of vaccine candidates 2,3 . A paper published by Wouters et. al 3 has presented an overview of challenges in production, affordability, allocation, and deployment of COVID-19 vaccines globally. The logistical and administrative challenges related to the identi cation of individuals in priority groups, digital tools to manage appointments and send reminders for second doses of vaccine were highlighted. To overcome these challenges most countries have mandated pre-registration using specially designed web portals.
The websites designed for registration are the single window of contact between the general population who want to get vaccinated and who are un-decisive for getting vaccinated, thus the vaccine registration portals should be provided to educate individuals on the bene ts of getting vaccinate, possible side effects, eligibility information should be mentioned, should be accessible through a range of devices, available in local languages, should include Frequently Asked Questions page, including contact details for more information or chatbot for any queries, able to explain the process of vaccination, direct individuals to nearest vaccination centre.
The objective of this study was to evaluate the COVID19 Vaccine Registration website for Quality of Information, Ease of Navigation and Interaction, and usability across various devices and formats, to enable informed decision making amongst the general population.

Methods
The objective of this survey was to evaluate the COVID-19 vaccine registration website, thus we used the following inclusion criteria c. National COVID-19 vaccination portals using English as one of the languages and is visible on Google search within the rst two pages of Google search after entering search terms The exclusion criteria for this study was a. following counties COVID-19 vaccination portal were identi ed for assessment and listed in Table 2. Section assesses the quality of information on treatment choices with 7 questions. The third Section with the Last question rates the overall rating of the publication based on the answers to all the previous 15 questions. The DISCERN rating scale for each question is 1 to 5, where 1 = de nite NO, 3 = partially and 5 = de nite Yes.
2. The QUEST tool 6 consists of 6 items with a subitem, Authorship, Attribution, Con ict of Interest, Complementarity, Currency, Tone, the Attribution has Subitem of "type of study". Each item is weighted on two factors: (i) how critical the item is to the overall quality of the article, established by a preliminary analysis of a sample of websites, and (ii) consideration of the criterion's ethical implications. Weight for Authorship is 1 X the score, Attribution is 3 X the score, Type of Study is 1 X the score, Con ict of Interest is 3 X the score, Complementarity is 1 X the score, Currency is 1 X the score, and for Tone is 3 X the score. The QUEST rating scale for each item is 2 to 0, where 2 = Highest score, 1 = Median score and 0 = lowest score For understanding usability across various devices and formats we used the Google test for mobile-friendliness.
The setting of Context to evaluation The study objective was to evaluate the Vaccination portals for the information provided by governments across UN Member states from a perspective of the general population and how the COVID-19 vaccination portals are equipped to help in informed decision making.
The DISCERN tool and QUEST tool were used for assessment as the DISCERN tool covers reliability, and quality of information about treatment choices, while QUEST tool covers ease of use, concision and comprehensiveness Thus, the authors had series of discussion and developed a standardised guideline to interpreting the questions and set in context-speci c to the study. The same is presented in Table 3. • Does the website say clearly about the advantage of getting vaccinated?
• Does it mention any adverse effects of vaccination?
• Is there any mention about the compensation or directs to appropriate authority in case of an adverse event?
4. Is it clear what sources of information were used to compile the publication (other than the author or producer)?
Hint: * Check whether the main claims or statements made about treatment choices are accompanied by a reference to the sources used as evidence (e.g. a research study or expert opinion) * Look for a means of checking the sources used such as a bibliography/reference list or the addresses of the experts or organisations quoted Look for the source mentioned under the above-mentioned topics, Like studies conducted within the country by the regulatory body or studies published by manufacturer cited 5) Is it clear when the information used or reported in the publication was produced?
Hint: Look for * dates of the main sources of information used to compile the publication * date of any revisions of the publication (but not dates of reprinting) * date of publication (copyright date) Rating note: The hints are placed in order of importance-to score a full '5' the dates relating to the rst hint should be found Look at the mention of revision of the information.
Speci cally, when the gap between the rst and second dose was increased for some vaccines, is the document supporting changes cited and date the change implemented 6) Is it balanced and unbiased?
Hint: Look for * a clear indication of whether the publication is written from a personal or objective point of view * evidence that a range of sources of information was used to compile the publication (e.g. More than one research study or expert)* evidence of an external assessment of the publication. Be wary if * the publication focuses on the advantages or disadvantages of one particular treatment choice without reference to other possible choices * the publication relies primarily on evidence from single cases (which may not be typical of people with this condition or responses to a particular treatment) *the information is presented in a sensational, emotive or alarmist way Here our focus should be on looking balance between the explanation given on the safety pro le and the bene ts of the vaccine.
Are bene ts overtly stated while details about safety are underwritten?
7. Does it provide details of additional sources of support and information?
Hint: Look for suggestions for further reading or details of other organisations providing advice and information about the condition and treatment choices Has the website provided a link to read in detail about the studies that were used as evidence to approve the use of the vaccine in-country, look for CTRI registration details 8. Does it refer to areas of uncertainty?
Hint: * Look for discussion of the gaps in knowledge or differences in expert opinion concerning treatment choices * Be wary if the publication implies that a treatment choice affects everyone in the same way (e.g. 100% success rate with a particular treatment) Look for the information that describes the use of vaccines to control the pandemic, look if the portal is making statements like "Vaccines will provide you complete protection against COVID-19" or "You don't need to use the mask after vaccination" such statements should be a sign of poor quality Section 2 How Good Is the Quality of Information on Treatment Choices? Focus on information provided regarding various vaccines supplied in the country 9. Does it describe how each treatment works?
Hint: Look for a description of how a treatment acts on the body to achieve its effect Look for information on how vaccine helps to build your immunity against COVID-19, is the information given in simple language, will the layperson understand the information easily, are any   (Table 3), by developing a consensus document to refer to in case of confusion.
In the results section, we are present the average score across all the member state for all the questions in DISCERN tool along with descriptive analysis on the reason for some member states scoring high while some scoring less. For the QUEST tool for all the seven areas the scores are presented in the table and descriptive analysis is presented From our analysis, the Republic of South Korea, Indonesia and the Czech Republic were the three countries that scored the highest overall score of 5 in DISCERN tool, Czech Republic and the United Arab Emirates scored a maximum score of 28 on the QUEST tool. Overall across both the tools Czech Republic's website for COVID-19 registration was found to be having the highest Quality of Information, Ease of Navigation and Interaction and was found to be mobilefriendly.
Vaccination Registration portals of all the 12-member state assessed were found to be mobile-friendly.

DISCERN Tool analysis
For question 1 (Are the aims clear?); all member states assessed except Bangladesh were rated a score of 5 on the Likert scale. The mean score across all the member states was 4.8 (SD ± 0.87) Bangladesh scored 2 because the website didn't mention statistics on vaccination drive in the country "Providing information about the vaccination programme", the homepage did provide the link for registration and explained the process of getting vaccinated. For question 15 (Does it provide support for shared decision-making?); the average score across all the member states was 3.4 (SD ± 1.51). Canada, Indonesia, Czech Republic and the United Arab Emirates scored 5 on a Likert Scale of 0 to 5 as they provided toll-free landline number to connect with GP on for any further questions, listed out do's and don'ts, asked individuals to consult to a general practitioner for more information. Republic of South Korea, Australia scored 4 on the same Likert scale, Nigeria, India and Bangladesh scored 3. Kazakhstan scores 2 while South Africa and Argentina scored 1 as lesser information was provided to connect for further queries.
Overall on DISCERN tool, the Czech Republic scored a perfect 5 across all the 15 questions, the average score across all the member states for DISCERN tool was 3.6 (SD ± 1.63). The UN Member state-speci c scores are added in the supplementary material.

QUEST Tool Analysis
The maximum score on the QUEST tool was 28 across all 7 parameters. The Czech Republic and the United Arab Emirates scored 28(100%), followed by Indonesia, Kazakhstan and the Republic of South Korea. South Africa scored the lowest score followed by India. The detailed score is presented in Table 4 across all the 12-member states Quality of the information on treatment choices -The vaccination portal should address the questions on the e cacy of vaccines available, safety pro le, the effect of the vaccine on quality of life, voluntarily and scope of differing from getting vaccinated, information of current availability of vaccines to speci c groups and populations.
Ease of use -The home screen of the vaccination portal should be designed be to intuitive to enable individuals to browse the content throughout the website, the font size, font style, animations and theme should be legible on small screens ( less than 6 inches diagonal size) and large screens (more than 10 inches diagonal size). Animations should be minimum, information should be available in multiple languages (English and regional language at least), availability of specially-abled friendly tools.
Concision and Comprehensiveness -The portal should include concise information speci c to COVID-19 vaccination avoiding regional, non-speci c messages on the portal.
We ran a correlation analysis to explore the relationship between COVID-19 vaccination rate and the DISECRN and QUEST scores. The data on rate of vaccination form Mathieu et a 8 . The rate of individuals administered at least one dose of vaccine per million population as on 27th May 2021 was abstracted and scatter plots were plotted against DISCERN Score and QUEST Score.
The Pearson Correlation Co-e cient value for relation between vaccination rate per million and DISCERN score was 0.61 (P value 0.05), Fig.1 presents the scatter plot.
The Pearson Correlation Co-e cient value for relation between vaccination rate per million and QUEST score was 0.51 (P value 0.10), Fig. 2 presents the scatter plot.
The Scatter plot added a limited value as the rate of vaccination is in uenced by variety of factors other than quality of information.
The current study to our knowledge is the rst of a kind to assess the quality of the COVID-19 vaccination website from a consumer health perspective and evaluate the information for reliability, quality on treatment choices, ease of use, concision and comprehensiveness. The study has its limitations as the information on the websites is updated due to the dynamism of COVID-19 treatment and vaccination guidelines, the study might not be replicable with similar results. .

Conclusions
Our study concluded that on DISCERN tool overall reliability score on the Likert scale of 0 to 5 was 4 (SD ± 1.28). On Quality of information regarding treatment choices, the average score was 3.4 (SD ± 1.67). The QUEST tool on ease of use, concision and comprehensiveness demonstrated an average score of 18.1 (SD ± 8.3). The vaccination portal of the Czech Republic scored the maximum score across DISCERN and QUEST tool. UN Member states of India, Bangladesh, Nigeria, South Africa scoring lower than their counterparts were due to a lack of clear information on treatment choices, limited information on the safety and e cacy of vaccines.
The low score indicates lack of Consumer Health Informatics hindsight, while developing the portal. However, as these websites are the only portals for registration, and effective and strategic placement of critical information, which would enable general population to develop trust on the vaccination drive and clear their doubts is a key to deter vaccine hesitancy and improve vaccination rate. Countries like India, Bangladesh, South Africa, Nigeria have huge population to vaccinate and a reliable and quality vaccine registration portal should be the rst priority of the administration.

Competing interests
The authors declare no competing interests. Figure 1 Scatter plot for correlation between Vaccination rate and DISCERN score Scatter plot for correlation between Vaccination rate and QUEST score

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.