Sample characteristics. We surveyed U.S. residents using Prime Panels which samples from a population similar to the U.S. population76. The surveys were conducted on May 24, 2022 (N = 400) and November 11, 2022 (N = 615). Supplementary Table 1 provides information on the sample demographics with the self-reported ages, education level, gender, race, ethnicity, political affiliation, COVID-19 infection history, and COVID-19 vaccination status.
U.S. residents’ risk perceptions related to design of RRT. The design of the RRT was based on studies about what would be understandable and useful to a large number of people70–73. Table 1 displays results of questions about risk perception in the sample the RRT was tested on.
The large number of people who believe vaccination is equally or more risky than infection underscores the need for an understandable educational tool to communicate relative risk. Further, many participants dramatically overestimated the number of U.S. citizens who have died as a result of vaccination. Figure 3 shows non-vaccinated participants are much more likely to think vaccination is equally or more risky than infection, May, \({\chi }^{2}\)(9, N = 400) = 149.05, p < .001, W = .61; November, \({\chi }^{2}\)(9, N = 615) = 189.75, p < .001, W = .56.
Most participants gave numerical risk estimates (due to vaccination and infection) much higher than those estimated in peer-reviewed literature77 (Fig. 4). We asked “If you receive a COVID-19 vaccine/booster in the future, what do you think your percent risk of a serious adverse reaction would be? An example of a serious adverse reaction is an allergic reaction requiring treatment in a hospital. Do not include your risk of common side effects such as fatigue” (Fig. 4a and 4b) and “If you were infected with COVID-19 (without any vaccination), what do you think your percent risk of hospitalization from COVID-19 would be?” (Fig. 4c and 4d).
After asking residents for percent risk estimates, we asked “If you got a COVID-19 vaccine, how concerned would you be about a serious adverse reaction to the vaccine?” and “If you were infected with COVID-19, how concerned would you be about getting hospitalized?” The level of concern participants attached to a given risk estimate, such as 50% risk of hospitalization, varied widely (Fig. 4). Many of the participants who estimated their risk of hospitalization if infected at 50% would be “not at all”, “slightly”, or “somewhat” concerned if infected as indicated by the shading in Fig. 4. Risk estimates of 50% are common, and the frequency of estimates of 50% is conveyed by the width of the bars in Fig. 4 (50% Vaccination risk: May, n = 77, 18.5%; November, n = 115, 14.2%; 50% Infection risk: May, n = 77, 19.3%; November, n = 115, 18.7%). Prior research that found that people often give 50% probability estimates to indicate “I don’t know” or “I’m uncertain” rather than to indicate that half the time it will occur54,55.
Although the median estimates for risk due to infection (May 26%; November 38%) were higher than median estimates for risk due to vaccination (May 20%; November 15%), the median responses were the same order of magnitude for both risks. In contrast, for an unvaccinated 40-year-old, the chance of hospitalization from recent COVID-19 infection (1.42%) is roughly 6000 times as large of the chance of a severe allergic reaction to the Pfizer COVID-19 vaccine (0.00021%)77,78.
Complacency about the risk of a disease is associated with lower vaccination uptake25, and the risk estimates related to COVID-19 infection on the RRT are much lower than the overestimates most people gave before seeing the tool (Fig. 4). The RRT was designed to help people understanding the severity of infection by comparing the risk due to infection to other familiar risks such as driving 1000 miles.
Risk perceptions are correlated with vaccination intent. We quantified the impact of risk perception on vaccination intent while controlling for other variables commonly associated with vaccination intent (see Fig. 2 for intent question and control variables). The multivariable ordinal logistic regression predicts vaccination intent, May, \({\chi }^{2}\) (11, N = 400) = 299.95, p < .001; November, \({\chi }^{2}\)(11, N = 615) = 386.83, p < .001. We chose the predictors in the initial full model and the reduced model presented based on the procedure described in the methods. The adjusted odds ratios shown in Fig. 5 tell us how many times as likely it is for someone with a given trait to be in one level higher in vaccination intent than a person in the associated reference category.
Overall, participants’ opinions about the risks and benefits of COVID-19 vaccination are correlated with future vaccination intent. Participants who said that vaccination and infection are equally risky were less sure about receiving a vaccine in the future than someone who said infection is riskier than vaccination (May aOR = 0.31, 95% CI [0.17, 0.56]; November aOR = 0.46, 95% CI [0.28, 0.75]). Similarly, participants who said vaccination is riskier than infection were less likely to intend to get vaccinated in the future than someone who said infection is riskier than vaccination (May aOR = 0.13, 95% CI [0.06, 0.30]; November aOR = 0.51, 95% CI [0.29, 0.89]). Further, participants who did not know what was riskier were less likely to intend to get vaccinated in the future (May aOR = 0.41, 95% CI [0.20, 0.85]; November aOR = 0.50, 95% CI [0.28, 0.88]). Consistent with these findings, participants who were concerned about severe adverse reactions to vaccination were less likely to intend to get more vaccination shots (May, aOR = 0.76, 95% CI [0.63, 0.91]; November, aOR = 0.71, 95% CI [0.61, 0.83]). Participants who were concerned about COVID-19 infection were more likely to intend to be vaccinated in future (May, aOR = 1.78, 95% CI [1.50, 2.12]; November, aOR = 1.70, 95% CI [1.46, 1.97]). Participants who did not think the vaccine reduces the risk due to COVID-19 infection were less likely to intend to be vaccinated (May aOR = 0.17, 95% CI [0.09, 0.32]; November aOR = 0.17, 95% CI [0.10, 0.27]). Similarly, participants who were not sure if the vaccine reduces risk were less likely to intend to be vaccinated as those who thought the vaccine reduces risk (May aOR = 0.34, 95% CI [0.19, 0.63]; November, aOR = 0.25, 95% CI [0.15, 0.42]).
The RRT succinctly conveys that infection is more likely to cause hospitalization and death than vaccination for all ages modeled, that vaccination reduces the risks due to infection substantially, and that unvaccinated adults face more risk from infection than people are typically willing to accept as a part of daily life. The odds ratios shows that beliefs about relative risk are more strongly associated with intent than education (which was only statistically significant in November) as well as age, gender, ethnicity, and race (which were removed from the models due to lack of statistical significance).
RRT impacted beliefs about relative risk. Respondents were asked to express their opinion on which posed a higher risk for them: the COVID-19 vaccination or infection. These opinions were collected both before and after their visit to the CDC website or RRT tool (for questions, see Fig. 2, and for responses, see Fig. 6). We used a McNemar-Bowker test79 to analyze symmetry in dependent responses.
For respondents who viewed the CDC website in May, the global test was non-significant (p = .681) indicating symmetric responses (i.e., the number of respondents who changed from one option to another was not significantly different than the converse). For respondents who viewed the RRT tool in May, the overall test was significant, and there was a medium effect size (p = .001, g = .23, 95% CI [.17, .35]) indicating a lack of symmetry80. This effect can be seen in the non-symmetric responses pre and post exposure to the RRT in Fig. 6. Post-hoc pairwise symmetry tests were conducted to determine between which subgroups there was a significant change, and adjusted p-values were calculated using the Benjamini-Hochberg procedure. In May, the pairwise tests showed a significant change from “COVID-19 infection and vaccination are equally risky” to “COVID-19 infection” (p = .010) as well as from “I don’t know” to “COVID-19 infection” (p = .010). In November, the global test for symmetry was statistically significant, and there was a medium effect size (p < .001, g = .16, 95% CI [.11, .23]). The post-hoc tests indicated that there were significant changes from “COVID-19 infection and vaccination are equally risky” to “COVID-19 infection” (p = .043), from “I don’t know” to “COVID-19 infection” (p = .012), and from “Vaccination is more risky” to “COVID-19 infection and vaccination are equally risky” (p = .043). See Supplementary Table 2 for all p-values.
Figure 6 shows the distribution of responses before and after the educational RRT intervention. In May, 48.8% (95% CI [43.9, 53.6]) of participants thought infection was riskier than vaccination before seeing the RRT, and this changed to 62.0% (95% CI [55.3, 68.3]) post-exposure. In November, 54.5% (95% CI [50.5, 58.4]) thought infection was riskier than vaccination before seeing the RRT, and this changed to 61.6% (95% CI [57.7, 65.4]) post-exposure.
RRT increased intent to accept vaccination. Before and after seeing information from either the CDC or RRT, we asked participants to rate their future intent to be vaccinated (Fig. 7). People who were already up to date on vaccinations were asked if they would get an additional booster if it was recommended.
After seeing the CDC information, the percentage of people who claimed they would “definitely not” be vaccinated decreased from 25.3% (95% CI [21.2, 29.7]) to 16.1% (95% CI [11.6, 22.0]). No one changed their mind to “No, definitely not” after seeing CDC information about side effects and efficacy. The McNemar-Bowker test indicated a global lack of symmetry, and there was a medium effect size (p = .004, g = .23, 95% CI [.17, .36]). Movement between subgroups “No, definitely not” and “Unsure, lean no” was not symmetric (p = .004) with people changing their mind from “No, definitely not” to “Unsure, lean no” but not vice versa.
Figure 7 also shows participants’ intent to be vaccinated before and after seeing the RRT. Before seeing the RRT in May, 25.3% (95% CI [21.2, 29.7]) would definitely not be vaccinated, and after seeing the RRT, this decreased to 16.3% (95% CI [11.9, 22.0]). Before seeing the RRT in November, 25.2% (95% CI [21.9, 28.8]) would definitely not be vaccinated, and after seeing the RRT this decreased to 15.4% (95% CI [12.8, 18.5]). According to a global McNemar-Bowker test, the changes in responses before and after seeing the RRT in May were not symmetric, and there was a large effect size (p < .001, g = .28, 95% CI [.20, .39]). In May, responses were not symmetric with movement from “No, definitely not” to “Unsure, lean yes” (p = .013) and “Unsure, lean no” to “Unsure, lean yes” (p = .009). In November, the responses before and after seeing the RRT were not symmetric, and there was a medium effect size (p < .001, g = .22, 95% CI [.18, .29]). There were significant changes from subgroups “No, definitely not” to “Unsure, lean yes” (p < .001) and “No, definitely not” to “Unsure, lean no” (p < .001) and from “Unsure, lean no” to “Unsure, lean yes” (p < .001). Supplementary Table 4 includes all adjusted p-values.