RECRUITMENT AND RETENTION
From February 11th, 2022 to June 30th, 2022, 2,045 participants were enrolled and randomly assigned to the control and intervention groups. After excluding 1,280 participants who were lost to follow up, responses from 748 participants were included in final analysis. (Figure 1).
PARTICIPANT CHARACTERISTICS
Across all five study groups in the three locations, the baseline characteristics between the control and intervention groups were mostly comparable (Tables S1 and S2). We noted an overrepresentation of minority subpopulation (i.e., Filipinos and Indonesians) in the Hong Kong and Singapore groups.
FINDINGS FROM THE Randomised Controlled Trial
Child group outcomes
In the Thailand child group, results from a one-way comparison using Fisher’s exact test showed that participants in the intervention group were less likely to report decreases in vaccine effectiveness: “vaccines are effective” [Intervention: 4.3% vs. Control: 17%, P = 0.023], “vaccines are effective regardless of manufacturers” [14% vs. 55%, P < 0.001], and “vaccines are effective against all variants” [8.7% vs. 48%, P < 0.001] compared to the control group (Table 1). In the Hong Kong child group, participants in the intervention group were less likely to report decreases in confidence regarding vaccine effectiveness [10% vs. 26%, P = 0.034], safety [12% vs. 21%, P = 0.047] and effectiveness in reducing severe conditions [14% vs. 21%, P = 0.031] (Table 1). In the Singapore child group, we did not detect any significant findings from primary outcomes due to high vaccination coverage rates and small sample size. Regarding secondary outcomes, the Fisher’s exact test showed that more parents in the intervention group had increased perceived risk of infection (“my child might get COVID-19” [47% vs. 24%, P = 0.015]) post-intervention compared to the control group (Table S3).
In the regression model, chatbot users from the Thailand child group were more likely after the intervention to have increased confidence in vaccines’ importance [OR=2.40 (95% CI: 1.34 to 4.32)], effectiveness in reducing severe conditions [OR=2.07 (1.23 to 3.48)], effectiveness regardless of manufacturers [OR=4.04 (2.87 to 5.60)], and effectiveness against all variants [OR=3.21 (2.22 to 4.66)] compared to the control group; however, the intervention group was less likely to have improved vaccine acceptance after the intervention (“has your child received/do you intend for your child to receive a COVID-19 vaccine?”) [OR=0.66 (0.45 to 0.96)] compared to the control group (Table 3).
Senior group outcomes
In the Thailand senior group, a comparison using Fisher’s exact test showed that participants in the intervention group were less likely, after the intervention, to report a decrease in vaccine confidence related to effectiveness in reducing severe conditions [12% vs. 21%, P = 0.024] and misinformation awareness related to COVID-19 vaccines’ completion of clinical trials [3.4% vs.18%, P = 0.026] than the control group (Table 2 and S4). In the Hong Kong senior group, Fisher’s exact test showed no significant findings on either the primary or secondary outcomes.
The regression model showed the Thailand senior intervention group was more likely to show improvement in debunking misinformation against COVID-19 vaccines’ completion of clinical trials [OR=2.33 (1.16 to 4.71)] compared to the control group. However, chatbot users were less likely to have increased confidence in vaccine safety [OR=0.63 (0.41 to 0.96)] and effectiveness regardless of manufacturers [OR=0.57 (0.36 to 0.88)] after chatbot use compared to the control group (Table 3). In Hong Kong, the intervention group was more likely to show improvements after the intervention in vaccine acceptance [OR=3.26 (2.53 to 4.21)], and confidence in vaccine importance [OR=1.87 (1.12 to 3.12)] and effectiveness [OR=1.72 (1.11 to 2.64)] compared to the control group (Table 4).
A set of sensitivity analyses were conducted to compare regression results both with and without the Filipino and Indonesian populations in our Hong Kong and Singapore groups (Figures S2-S3). The results showed no statistically significant differences.
FACTORS OF VACCINE CONFIDENCE AND ACCEPTANCE
Sociodemographic factors
Chatbots were found to be significantly more effective at improving vaccine confidence and acceptance among vaccine-hesitant people who are minorities (i.e., non-Thai in Thailand and non-Chinese in Hong Kong and Singapore) and those who have lower education levels (i.e., below college level). Specifically, in the child groups, being a minority was associated with higher odds of an improved belief in vaccine effectiveness [OR: Thailand = 12.91 (3.56 to 46.78); Hong Kong = 2.73 (1.11 to 6.7)], effectiveness regardless of manufacturers [OR: Thailand = 2.99 (1.28 to 6.97)], and safety [OR: Hong Kong = 5.86 (2.01 to 16.1); Singapore = 10.46 (1.14 to 95.79)] (Tables 3-5).
In the Hong Kong child group, parents with a college or above education level showed lower odds of experiencing improved perceptions of vaccine importance [OR=0.26 (0.14 to 0.48)] and effectiveness in reducing severe conditions [OR=0.34 (0.18 to 0.64)], as well as vaccine acceptance [OR=0.33 (0.22 to 0.48)]. Likewise, in the Hong Kong senior group, respondents with a college or above education level showed lower odds of experiencing improved perceptions of vaccine importance [OR=0.31 (0.18 to 0.55)], safety [OR=0.18 (0.11 to 0.29)], and effectiveness [OR=0.41 (0.26 to 0.67)] (Table 4).
Misinformation
Respondents who were better at identifying COVID-19 vaccine misinformation in the baseline survey were more likely to have improved vaccine confidence and acceptance. In the Hong Kong child group, parents who correctly identified the misinformation: “COVID-19 vaccines cause infection" in the pre-intervention questionnaire were more likely to have improved beliefs in vaccine importance [OR=2.24 (1.39 to 3.62)], safety [OR=6.09 (3.51 to 10.55)], effectiveness [OR=1.67 (1.09 to 2.56)], and acceptance [OR=1.77 (1.32 to 2.36)] (Table 4). In the Singapore child group, parents who correctly identified the misinformation: “COVID-19 vaccines cause genetic change" were more likely to have improved beliefs in vaccine importance [OR=7.28 (2.35 to 22.54)], safety [OR=1.86 (0.73 to 4.76)], and effectiveness [OR=5.24 (1.81 to 15.16)] (Table 5).
Risk perceptions
Respondents with higher risk perceptions were less likely to improve in their vaccine confidence and acceptance compared to those with lower risk perceptions (Tables 3-5). In the Thailand child group, high risk perception of the severity of COVID-19 disease was negatively associated with increases in beliefs of importance [OR=0.62 (0.45 to 0.86)], vaccine safety [OR=0.7 (0.53 to 0.92)], effectiveness in reducing severe conditions [OR=0.71 (0.54 to 0.93)], effectiveness regardless of manufacturers [OR=0.65 (0.55 to 0.78)], and effectiveness regardless of all variants [OR=0.67 (0.55 to 0.82)]; in the Singapore child group, higher risk perception of COVID-19 infection was negatively associated with vaccine acceptance [OR=0.42 (0.26 to 0.68)].
Sub-communities: Filipinos and Indonesians in Hong Kong and Singapore
Similarly, among the Filipino and Indonesian sub-communities in Hong Kong and Singapore – who are mostly residents on a working visa,33-35 higher risk perceptions of COVID-19 disease severity were negatively correlated with beliefs in vaccine effectiveness for reducing severe conditions [OR=0.6 (0.43 to 0.83)] (Table 5). Conversely, higher risk perceptions were also correlated with greater beliefs in vaccine importance in these sub-communities [OR=1.35 (1.07 to 1.71)] (Table 5).
RE-AIM FRAMEWORK: EVALUATION OF THE INTERVENTION
Results from RE-AIM evaluations on the chatbot intervention are presented in Table S6. In brief, Hong Kong and Singapore’s D24H chatbot reached a total of 185 targeted users during the study. Thai ChatSure, which was further along in development compared with the D24H chatbot and had already been publicly accessible, reached 40,613 users since its launch in June 2021 and 1,495 users during the study period. Both chatbots were supported by existing studies that demonstrated chatbots’ potential for combating infodemic via prompt information dissemination53 and improving vaccination intention through enhanced knowledge and self-efficacy.57 Our formative data on chatbot users’ adaptions to the intervention suggested considerable interest in the public toward vaccine-related chatbots, as corroborated by the proportion of chatbot users who agreed or strongly agreed with the question: “I intend to use the chatbot again” [Thailand = 87%; Hong Kong = 73%; Singapore = 82%]. The Thai Ministry of Health brought ChatSure to the public specifically to combat COVID-19 vaccine-related misinformation; the D24H chatbot is currently being expanded to cover other vaccines, such as the HPV vaccine, so it may serve as a scalable intervention for existing vaccination campaigns to enhance online engagement with the goal of increasing vaccine confidence.