Nudge-Based Misinformation Interventions are Effective in Information Environments with Low Misinformation Prevalence

doi:10.21203/rs.3.rs-3736230/v1

Download PDF

Article

Nudge-Based Misinformation Interventions are Effective in Information Environments with Low Misinformation Prevalence

https://doi.org/10.21203/rs.3.rs-3736230/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Nudge-based misinformation interventions are presented as cheap and effective ways to reduce the spread of misinformation online. However, despite online information environments typically containing low volumes of misinformation, most studies testing the effectiveness of nudge interventions present equal proportions of true and false information. As the effectiveness of nudges can be highly context-dependent, it is imperative to validate the effectiveness of nudge-based interventions in environments with more realistic proportions of misinformation. The current study (N = 1387) assessed the effectiveness of a combined accuracy and social-norm nudge in simulated social-media environments with varying proportions of misinformation (50%, 20%, and 12.5%). The nudge intervention was effective at improving engagement behavior in all conditions, including those with a lower proportion of misinformation. These findings provide ecologically valid support for the use of nudge-based interventions to counter misinformation propagation on social-media platforms.

Biological sciences/Psychology

Biological sciences/Psychology/Human behaviour

Misinformation—defined here as false or misleading information presented as accurate, regardless of intent—has the capacity to negatively impact people’s beliefs and behaviors [1]. Behaviors driven or justified in part by misinformation, such as vaccination refusal, can not only detrimentally impact an individual (e.g., increasing the risk of contracting potentially life-threatening illnesses), but also others (e.g., increasing the potential spread of viruses to vulnerable populations) and society more broadly (e.g., placing increased pressure on the health-care system [2]). Due to the threats posed by misinformation, developing and implementing interventions that reduce people’s susceptibility to misinformation has become a key focus of both research and policy [1, 3, 4].

To date, several psychologically-based misinformation interventions have received sound empirical support (for a synthesis of currently recommended interventions, see Kozyreva et al. [5]). For example, providing corrective information that directly counters a piece of misinformation (i.e., “debunking”) can significantly and meaningfully reduce belief in the targeted false information [6]. However, although targeted interventions can be highly effective, the advent of social media has made detecting and directly counteracting misinformation increasingly difficult, and in many cases impossible [7]. As such, in recent years there has been an increased focus on developing and implementing generalized misinformation interventions that are easily scalable to social-media environments [5, 8, 9]. Many of these interventions are based on nudge theory, which posits that small changes in choice architecture can meaningfully impact decision-making processes [10]. In the realm of misinformation, nudge-based interventions typically attempt to reduce misinformation sharing that may occur due to inattentiveness to information veracity [11] by priming people to consider (1) the accuracy of encountered information (i.e., accuracy nudges [12, 13]) or (2) the attitudes or behaviors of others as they pertain to misinformation sharing (i.e., social-norm nudges [14, 15]). Nudge-based misinformation interventions are proposed to be effective because they draw people’s attention to the importance of veracity, subsequently increasing the weight placed on veracity as a criterion during decision-making processes (in line with the limited-attention utility model [11]).

In experimental settings, nudge-based misinformation interventions have generally shown to have a beneficial, though small, impact on engagement behavior through either directly reducing intent to share false information or improving “sharing discernment” (i.e., increasing the proportion of true relative to false information participants report they would share [8, 12, 14–16]). However, despite these positive findings, studies assessing the effectiveness of nudge-based misinformation interventions do not always appropriately consider the structure of the social-media information environment. Specifically, (1) the proportion of false information is often artificially high (e.g., 50% of the claims presented), (2) participants are often exposed to only verifiable (i.e., objectively true or false) information, often in the form of news headlines, and (3) participants are often required to actively appraise whether or not they would engage with each item (e.g., headline). In contrast, the quantity of misinformation people are exposed to on real-world social-media platforms is typically small compared to the amount of true or non-verifiable (e.g., personal or opinion-based) information [17]. Additionally, the volume of information people are exposed to on social media exceeds what they are able or inclined to critically appraise [11, 18].

Research outside of the field of misinformation has consistently shown that nudges can be highly context-specific and quick to decay [10, 19]. Therefore, it is unclear whether the effect of nudge-based interventions observed in typical experimental settings would be observed in more realistic information environments. In fact, exploratory analyses by Roozenbeek et al. [16] suggest accuracy nudges may only be effective on the few posts immediately succeeding the nudge. If this is the case, nudge-based misinformation interventions may be less effective than believed, particularly when misinformation makes up a small proportion of content in the information environment. As the implementation of interventions, especially those which shift responsibility to the individual consumer, potentially have detrimental practical consequences if ineffective [20], the question of whether the effects of nudge-based interventions translate to more realistic information environments requires direct investigation.

We are aware of one recent study that assessed the efficacy of an accuracy-nudge intervention in a setting which included non-news-based posts in addition to true and false headlines [21]. Participants were presented with 72 posts, of which 48 were social posts (50% political, 50% apolitical), 12 were true headlines, and 12 were false headlines. The researchers found accuracy prompts led to a small improvement in sharing discernment, however, no improvement in liking discernment (in fact, liking of false posts was numerically greater than control in all accuracy-nudge conditions, in some cases statistically significantly so). Additionally, the accuracy nudge neither significantly reduced engagement with (or sharing of) false information, nor significantly increased engagement with (or sharing of) true information compared to the control condition. This suggests that nudging may have only limited positive impact on sharing discernment in environments with lower volumes of misinformation. However, in this study the proportion of each information type (i.e., true, false, and social) was kept constant across all conditions. As such, it is unclear whether the effectiveness of nudge interventions varies depending on the proportion of misinformation relative to true and non-news information.

The overarching aim of the current study was to assess the effectiveness of a scalable nudge-based intervention (specifically, a combined accuracy-prompt and social-norm intervention) in environments with varied proportions of misinformation. Across conditions, false headlines made up either 50% (40 false headlines, 40 true headlines), 20% (10 false headlines, 40 true headlines), or 12.5% (10 false headlines, 40 true headlines, 30 social posts) of total posts. To increase external validity, posts were presented in a mock feed using a realistic social-media simulator [22], and participants were not required to actively appraise all posts within the study (i.e., as with real social-media platforms, participants had the capacity to scroll past posts without reading them).

We specified the following pre-registered hypotheses: It was hypothesized that (1) there would be a main effect of the nudge intervention, such that the nudge would significantly improve engagement discernment. Further, and central to the current research question, it was hypothesized that (2) the effectiveness of the nudge intervention would depend on the misinformation proportion. As this is the first time this question has been empirically assessed, we did not explicitly pre-register a directional hypothesis. However, due to the context-dependent nature of nudges, and prior research suggesting the effectiveness of nudge-based misinformation interventions may decay quickly [16], if the effectiveness of the nudge intervention does differ across conditions with varying proportions of misinformation it was predicted that the nudge intervention would be significantly more effective when the proportion of false headlines was high (i.e., 50%) than in the conditions with a lower proportion of false headlines (i.e., 20% and 12.5%).

This study had a 2 × 3 between-subjects design with factors nudge intervention (present, absent) and misinformation proportion (50%, 20%, 12.5%). Each participant was therefore randomly assigned to one of six possible conditions. Given the study included both true and false headlines, the design was technically a 2 (presence of nudge) × 3 (misinformation proportion) × 2 (headline veracity) between-within-subjects design. In line with the pre-registered analysis plan, the primary dependent variable was engagement discernment (i.e., difference in engagement with true and false headlines). Discernment was chosen as the primary outcome variable to be consistent with prior research [12–14, 16, 23]. However, because the effectiveness of nudge interventions is often framed in terms of reducing engagement with false information, we also looked at the impact of the nudge on false and true headlines in isolation. This study was pre-registered at https://osf.io/ch7n9, and the study was approved by the Human Research Ethics Office at the University of Western Australia (reference number: 2019/RA/4/20/6423) and complied with all relevant guidelines and the Declaration of Helsinki. Informed consent to participate and publish was obtained from all participants included in the study.

Participants

An a-priori power analysis (using G*Power [24]) suggested a minimum sample size of 1269 (approximately 212 participants per condition) to detect a small effect of f = 0.10 at α = 0.05 and 1 – β = 0.90. To ensure adequate sample size following anticipated exclusions, we thus aimed to sample 250 participants per cell (1500 participants total).

A total of 1501 U.S.-based participants with at least one social media account (e.g., Facebook, Instagram, TikTok) were recruited via Prolific. Due to a misinterpretation of instructions, 56 participants terminated the experiment before completing the belief-rating questions at the end of the experiment. Given these participants completed the primary task, and consented to their data being used, their data was retained for the primary analyses. Participants were excluded in accordance with the following a-priori criteria: (1) Self-reported English proficiency “fair” or “poor” (n = 0); (2) self-reported lack of effort (n = 2); (3) completion time for the simulated feed portion of the study < 3 min for the 12.5% and 50% misinformation conditions or < 2 min for the 20% misinformation condition (n = 21); (4) post dwell time in the simulated feed equal to 0 ms for more than 10% of posts (n = 68); (5) median dwell time across all posts < 2000 ms (n = 65); (6) an identical response to > 80% of the belief ratings (n = 22). An additional six participants were excluded for completing the simulated feed portion of the study multiple times. Accounting for participants who met multiple exclusion criteria, 114 participants were excluded, resulting in a final sample size of N = 1387 (1335 of which completed the belief ratings; M_age = 41.39, SD_age = 14.14; age range 18–85; 736 females, 621 males, 26 non-binary individuals, 2 transgender men, 1 transgender individual, 1 participant who reported no gender). The number of participants per condition ranged from 221 to 250.

Materials

Nudge Intervention

The nudge comprised a combined accuracy and social-norm prompt. The prompt was modelled off prior research [12, 14, 15]. Specifically, participants in the nudge conditions received the following information: “It is important to consider the accuracy of posts when engaging with content online. In fact, it is widely accepted that spreading misinformation is wrong and can have a variety of negative outcomes for both individuals and societies. Indeed, a recent study found that more than 80% of U.S. adults think it’s very important to only share accurate content online. This was true for both Democrats and Republicans. As such, please consider the accuracy of the headline on the following page.” Participants were then presented with a single false headline (“Newborn becomes first to be named an emoji”; See Appendix A, Figure A1), which was identical across participants, and were asked to rate the headline’s accuracy on an 11-point scale from 0 (“Certainly false”) to 10 (“Certainly true”).

Posts

Dependent on condition, participants were presented with either (1) 40 true news headlines and 40 false news headlines (50% misinformation condition), (2) 40 true news headlines and 10 false news headlines (20% misinformation condition), or (3) 40 true news headlines, 10 false news headlines, and 30 social (i.e., non-falsifiable) posts (12.5% misinformation condition).

We initially selected 40 true and 40 false headlines from a pool of 60 false and 60 true headlines. False headlines were sampled from a range of sources such as [25], Snopes.com, and Politifact.com and true headlines were sampled from a range of reputable news sources such as NPR, BBC, and CNN. All true headlines were cross verified with at least one other news source to validate its veracity. In two pilot ratings, all 120 headlines from the initial pool were rated on believability and political leaning by a sample of N = 50 Prolific workers (24 male, 25 female, one non-binary; M_age = 38.96, SD_age = 12.69), and on shareability and “currentness” by a separate sample of N = 52 Prolific workers (26 male, 25 female, one non-binary; M_age = 41.10, SD_age = 12.72; see Supplement A for full details). Selected true and false headlines were approximately matched on political leaning, shareability, and currentness; believability was substantially lower for false than true headlines. Ten false headlines were selected to be the target headlines used in all conditions (and thus the main analyses), maintaining comparable believability, political leaning, shareability, and currentness to the full set of false headlines used in the 50% misinformation condition.

Each headline was paired with a relevant image and either a mainstream (for true headlines; e.g., BBC) or non-reputable (for false headlines; e.g., REALRAWNEWS.COM) real-world news source. For the target false headlines, a unique non-reputable source was used for each headline (i.e., 10 sources total). In the 50% misinformation condition, three additional false headlines (which served as fillers and were not included in the main analyses) were paired with each non-reputable source, resulting in four false headlines being paired with each non-reputable source. For the true headlines, four headlines were paired with each mainstream source (i.e., 10 sources total). Pairing of headlines to news sources was fixed across participants. See Fig. 1 for an example post.

Social posts used in the 12.5% misinformation condition were generated by the authors, and designed to mimic typical user-generated content people may encounter on social media (e.g., buy and sell, holiday posts). As with news-based posts, social posts were paired with a relevant image.

For each participant, each post was randomly paired with a unique post source (i.e., the source that ostensibly shared the post into the feed) with a generic username (e.g., k_hannam), and each post source had a randomly determined number of followers (sampled from a positively skewed distribution with M = 0, SD = 500, skewness = 10, truncated at 0). Some additional parameters were also determined probabilistically: Posts were associated with a small number of prior likes and shares (ostensibly from other users), and participants’ follower counts changed dynamically depending on their engagement with posts (e.g., liking, sharing). Parameters of the normal distributions from which values were randomly sampled are given in Table 1. Parameters were kept constant across post types, however, the specific values paired with each post were randomized across participants.

Table 1

Normal-Distribution Parameters Used to Determine the Quantity of Prior Engagements Associated with Each Post and the Impact of Post Engagements on Participant Follower Counts. Note, Distributions Were Truncated at 0 for Prior-Engagement Quantities but not Changes to Followers (Which Were Therefore Sometimes Negative).
	Quantity of Prior Engagements		Changes to Followers
	M	SD	M	SD
Likes	5	10	+ 2	2
Shares	1	3	+ 4	2

Procedure

Participants were initially provided with an ethics-approved information sheet and provided informed consent and basic demographic information. Participants were then shown instructions on how to interact with the simulated social-media feed and were randomly assigned to one of the six conditions. In the nudge conditions, participants were initially presented with the combined accuracy and social-norm nudge. Because engagement is, to a degree, incentivised on social media [26], before beginning the task participants also received the prompt “Remember: Engage as you would on social media and try to maximize your follower count!”, however, participant’s final follower count had no real-world (e.g., monetary) consequence. Participants were then presented with all posts, in a randomized order, in a feed format (i.e., all posts were displayed on a single page, requiring participants to scroll down the page to view subsequent posts). Participants had the option to like and/or share, as well as comment, on each post. Participants could also not engage with a post by scrolling past it. Engagement was recorded, and follower counts were updated (where pertinent) once participants scrolled past the post (i.e., it was no longer visible on screen). Once recorded, participants could not change how they interacted with a post; however, participants could “scroll back” and engage with posts they had previously not interacted with.

After completing the social-media simulation, participants were presented with a subset of the headlines again (without the news source or image), and were asked to rate their belief in each headline on an 11-point scale from 0 (“Certainly false”) to 10 (“Certainly true”). To maintain the true:false proportions used in the main task, participants in the 12.5% and 20% misinformation conditions were presented with all 10 false and 40 true headlines presented in the social-media simulation. Participants in the 50% misinformation condition were presented with a subset of 25 false (always including the 10 target false headlines) and 25 true headlines in one of 16 counterbalanced combinations. All headlines were displayed for a minimum of 3 s. Participants were then asked a single question regarding their political orientation (specifically, “Where would you position yourself politically from strongly liberal to strongly conservative?”); responses were recorded on a 7-point Likert scale from 0 (“Strongly liberal”) to 6 (“Strongly conservative”). Upon completing the study, participants were asked whether their data should be discarded due to lack of effort and were provided a debriefing, which explicitly stated the purpose of the study as well as the veracity of all the presented true and false headlines. Median completion time was approximately 20 minutes and participants were compensated £2.20 (approx. US$2.75) for their time.

Belief and political orientation were included as secondary variables and thus the results are not included in the main text. Please see Supplement B for complete output for the main analysis, and Supplement C for an overview of the supplementary analyses.

Due to the ordinal nature of engagement behavior, primary analyses were conducted using cumulative-link mixed-effects models using the clmm function of the ordinal R package [27]; all data visualizations were created using ggplot2 [28]. Because we were interested in overall main effects and interactions, rather than only main effects and interactions relative to the reference group, we conducted analyses of deviance [29] on the fitted models using the Anova.clm function of the R package RVAideMemoire [30]. Post-hoc comparisons were conducted using emmeans [31]. Data and the R script, Rmarkdown file, and Supplementary information for all analyses is available at https://osf.io/nztuk/.

Analyses focused on the 10 false and 40 true headlines presented in all conditions. Prior to statistical analysis, the factors nudge (present, absent) and headline veracity (true, false) were centred, and misinformation proportion (12.5%, 20%, 50%) was factor-coded. As per the pre-registration, the primary dependent variable of engagement was coded as an ordinal factor with four levels (0 = no engagement, 1 = like, 2 = share, 3 = like and share). Within the pre-registration we specified that the maximal random-effects structure avoiding convergence errors would be used for each analysis. However, a number of models failed to converge with participant and headline (i.e., item) included as random intercepts. Due to this, we ran separate analyses either including or excluding participant as a random intercept. Models including (only) participant intercept demonstrated a significantly better fit; thus, we focus on analyses with participant random intercepts in the following. Please refer to Supplement C for results of the analyses using the alternative random-effects structure.

We assessed the impact of the nudge on engagement with true and false headlines (i.e., engagement discernment) across the conditions with different misinformation proportions (i.e., 12.5% misinformation; 20% misinformation; 50% misinformation). Results of the ANODE are presented in Table 2, and average level of engagement across conditions is presented in Fig. 2.

Table 2

ANODE Results for Engagement Behavior
Fixed Effects	χ²	df	p
Misinformation proportion	19.47	2	< .001
Nudge	0.16	1	.692
Headline veracity	944.29	1	< .001
Misinformation proportion × Nudge	2.10	2	.350
Misinformation proportion × Headline veracity	25.44	2	< .001
Nudge × Headline veracity	50.95	1	< .001
Misinformation proportion × Nudge × Headline veracity	0.97	2	.616

There was a significant fixed effect of headline veracity, with engagement with true headlines on average higher than engagement with false headlines. There was no fixed effect of the nudge intervention, however, there was a significant nudge × headline-veracity interaction, with engagement discernment comparatively higher in the nudge than the no-nudge condition, in line with the first hypothesis. There was additionally a significant fixed effect of misinformation proportion. This effect was qualified by a significant misinformation-proportion × headline-veracity interaction, suggesting that how people engaged with true and false headlines varied across conditions with varying misinformation proportions. However, contrary to the second hypothesis, there was no significant misinformation-proportion × nudge interaction; nor was there a significant three-way interaction.

Impact of Nudge Intervention on Engagement Behavior

To assess whether the nudge intervention significantly reduced engagement with false posts or significantly increased engagement with true posts (or neither), we ran follow-up planned contrasts isolated to false and true headlines. Overall (i.e., collapsed across misinformation proportion conditions), the nudge significantly reduced engagement with false headlines, β = 0.23, SE = 0.09, z = 2.42, p = .016, however, had no significant effect on engagement with true headlines, β = −0.08, SE = 0.09, z = − 0.96, p = .336.

Although there was no significant misinformation-proportion × nudge interaction, for completeness we assessed the impact of the nudge on engagement discernment isolated to each misinformation-proportion condition. The nudge intervention improved engagement discernment across all three conditions (12.5% misinformation; χ²(1) = 16.49, p < .001; 20% misinformation; χ²(1) = 25.32, p < .001; 50% misinformation; χ²(1) = 10.50, p = .001). That said, follow-up analyses isolated to false and true headlines revealed that the nudge was associated with a marginally significant reduction in false-headline engagement in the 12.5% and 20% misinformation conditions (12.5%: β = 0.34, SE = 0.17, z = 2.06, p = .039; 20%: β = 0.33, SE = 0.16, z = 2.08, p = .037), but no significant reduction in false-headline engagement in the 50% condition, β < .01, SE = 0.16, z = 0.04, p = .969. Regarding true-headline engagement, there was no statistically significant effect of the nudge intervention in any of the misinformation-proportion conditions, all ps ≥ .094.

Supplementary analyses were additionally run to assess the effects of the nudge intervention and misinformation proportion isolated to sharing and liking behavior. The overall pattern of results was comparable across engagement types; however, when analyses were isolated to false and true headlines, the nudge decreased sharing, but not liking, of false headlines across the 12.5% and 20% misinformation conditions (all other contrasts were non-significant).

Impact of Misinformation Proportion on Engagement Behavior

Post-hoc analyses, Holm-Bonferroni corrected, were run to deconstruct the impact of misinformation proportion on engagement with false and true headlines, collapsed across nudge conditions. Engagement with false headlines was significantly higher in the 20% misinformation condition than either the 12.5% or 50% misinformation conditions (12.5%: β = 0.48, SE = 0.12, z = 4.16, p < .001; 50%: β = 0.35, SE = 0.11, z = 3.07, p = .004). The 12.5% and 50% misinformation conditions did not differ with regards to false-headline engagement, p = .256. In contrast, engagement with true headlines was significantly lower in the 12.5% misinformation condition than either the 20% or 50% misinformation condition (20%: β = 0.45, SE = 0.11, z = 4.27, p < .001; 50%: β = 0.35, SE = 0.11, z = 3.28, p = .002). The 20% and 50% conditions did not differ in terms of true-headline engagement, p = .319.

Does the Effectiveness of the Nudge Intervention Decay Over Time?

Although not pre-registered, we ran exploratory analyses to assess whether the effect of the nudge intervention decayed over the course of the study. To do so, we included headline-display order in the model as a continuous predictor (as a proxy for time), and analyses were conducted separately for each misinformation-proportion condition. There were no nudge × post-order, or nudge × headline-veracity × post-order interactions (all ps ≥ .073; see Supplement C, Tables C21-C23 for full results). As such, there is no strong statistical evidence for (linear) decay over the course of the study. However, we note that the difference in engagement with false headlines between the nudge and no-nudge control conditions does appear to numerically decrease over the course of the study in the 20% and 12.5% misinformation conditions (see Fig. 3; though the reverse pattern seems to arise for engagement with true information in the 50% misinformation condition).

The current study provides support for the use of nudge-based misinformation interventions across information environments with varied proportions of misinformation. In fact, and contrary to our predictions, the results suggest nudge-based interventions were equally effective (and numerically even slightly more effective) at improving engagement with true and false headlines (i.e., engagement discernment) in environments with lower misinformation prevalence compared to a “balanced” environment with 50% misinformation. As real-world social-media environments typically contain a low proportion of false information [17], these findings therefore provide ecologically valid support for the use of large-scale nudge-based misinformation interventions on social media.

Across both the 12.5% and 20% misinformation conditions, the positive effect of the nudge on engagement discernment was mainly driven by a reduced level of engagement with false headlines (relative to the no-nudge control). In these conditions, the nudge was associated with no meaningful change in engagement with true headlines. By contrast, in the 50% misinformation condition, the effect of the nudge on engagement was non-significant when isolated to either true or false headlines. This said, the numerical pattern of results, as well as results of statistical models without participant random intercepts, suggested that in this condition the nudge improved engagement discernment primarily by increasing engagement with true headlines, rather than reducing engagement with false headlines. As such, there is some tentative evidence that what drives the effectiveness of the nudge intervention may differ depending on the proportion of misinformation in the information environment. However, given the absence of a significant interaction between misinformation-proportion and nudge condition, potential differential effects of the nudge intervention must be interpreted with caution.

Beyond the impact of the nudge intervention, misinformation proportion and the number and type of posts also had broader effects on participants’ engagement with both true and false headlines. Specifically, level of engagement with false posts was significantly higher in the 20% than in either the 12.5% or 50% misinformation conditions, and supplementary analyses suggest this was not simply driven by the reduced number of headlines in the 20% condition (nor participants engaging more at the beginning of the study; see Supplement C). Notably, engagement with true headlines was also higher in the 20% misinformation condition compared to the 12.5% condition, but was similar in the 20% and 50% conditions. Due to the slightly different pattern of results across the misinformation-proportion conditions, we discuss them separately below.

Engagement with false, but not true, headlines was significantly higher in the 20% misinformation condition than the 50% misinformation condition. Exploratory analyses suggested this difference was primarily driven by higher engagement when the nudge was not provided (see Supplement B, Table B3). One potential explanation of this difference is that, without the explicit accuracy nudge, the comparatively high proportion of true posts in the 20% misinformation condition led to participants placing less weight on accuracy as a decision-making criterion. Although speculative, this is consistent with the limited-attention utility model proposed by Pennycook and colleagues [11, 13] and suggests that the use of stimuli sets containing unrealistically high proportions of false information may, in itself, prime accuracy. Given the relatively low prevalence of misinformation in real-world information environments, using a 50% true:false ratio may thus be lowering the observed effectiveness of nudge interventions in experimental settings. That is, in some cases (e.g., when headline veracity is clear), there may be minimal benefit of an explicit nudge intervention over the implicit accuracy prime conveyed by the high proportion of misinformation. This perspective may also partially explain why in some studies (e.g., [8]; the 50% misinformation condition in the current study), the nudge intervention had no meaningful impact on engagement with false information but did increase engagement with true information: If the large proportion of false information is priming accuracy, and engagement with posts people believe are false is generally low, false-post engagement may be close to the floor, providing little room to decrease further in response to a nudge. By contrast, engagement with true posts is often low for reasons unrelated to veracity [32, 33], and as such there may be comparatively more room for engagement with true posts to increase following a nudge intervention. However, additional research is needed to tease apart the comparative effect of implicit, context-driven, accuracy primes and explicit nudges on engagement behavior.

By contrast, reduced engagement in the 12.5% misinformation condition compared to the 20% condition was relatively equivalent across headline types and nudge conditions, and engagement with true headlines was also lower in the 12.5% than the 50% misinformation condition. The 12.5% condition was the only condition to include social posts alongside the news posts, and engagement with social posts was higher than with either true or false news headlines (see Supplement C, Table C26 and Figure C9). This suggests that although people may be somewhat inattentive to veracity on social-media platforms, they may also be somewhat disinterested in or unwilling to actively engage with news-based posts [32, 33]. Given non-news information makes up the bulk of content people are exposed to on social media [11, 17], people’s likelihood of engaging with news information may thus, on average, be lower than suggested by studies that only use news-based stimuli. Nonetheless, it is important to stress that the nudge intervention improved engagement discernment across all conditions, and the numeric pattern of results was similar across the 12.5% and 20% misinformation conditions. As such, although the engagement with news (mis)information may be lower in environments containing non-news information, nudge-based interventions appear robust to such variation.

Although the results are generally promising, there are some limitations of the current study that should be addressed in future research. Firstly, while the procedure mirrored prior research, the temporal distance between the provision of the nudge intervention and participants engaging with news headlines was brief. As such, although the nudge was effective in environments with a reduced proportion of misinformation when tested immediately, we cannot make claims about the enduring effectiveness of the intervention. In fact, when assessing the effect of the nudge intervention across headline display order (as a proxy for time), there was minimal numeric difference in engagement between the nudge and no-nudge conditions within the 12.5% misinformation condition by the end of the task (though we note that the reduction in nudge effectiveness was not statistically significant, and the effect of the nudge appears to endure longer than the effect observed in Roozenbeek et al. [16]). Although admittedly crude, this apparent trend calls for future research to directly assess the effectiveness of nudge-based interventions after a delay.

Secondly, the effectiveness of the nudge intervention was assessed on news headlines only. While headlines are a common focus of misinformation research, this limits our ability to generalise to other forms of accurate and inaccurate content [20]. Misinformation comes in many forms, and can be more ambiguous and subtle than the false headlines used here. Therefore, it would be beneficial to assess whether the current results generalize across a broader range of content types. Finally, despite testing the effectiveness of the nudge intervention in conditions with reduced misinformation, the proportions of misinformation used in this study remained somewhat higher than is the case in some social-media environments [17, however, cf. 34]. Although having even less false information would limit statistical power, the effectiveness of nudges may be reduced in these circumstances. Nonetheless, the current pattern of results suggest that nudge-based interventions are unlikely to negatively impact engagement even in information environments with negligible misinformation. Additionally, and crucially, within social-media environments that do contain non-negligible proportions of misinformation, nudges appear to lead to beneficial changes in engagement behavior.

Cumulatively, the current findings suggest that even when misinformation volume is low, large-scale nudge-based interventions are likely to lead to an immediate improvement in engagement behavior. Although additional research is required into the enduring effectiveness of the intervention over time and misinformation type, these findings provide theoretical support for the limited-attention utility model [11], and suggest nudge-based interventions are generally well suited to the online social-media environments they are intended to be implemented in.

Author Contributions:

L.H.B. – Conceptualization, Investigation, Software, Methodology, Data Curation, Formal Analysis, Visualization, Writing (original draft)

T.P. – Conceptualization, Methodology, Validation, Writing (reviewing and editing)

U.K.H.E. – Conceptualization, Methodology, Funding Acquisition, Supervision, Writing (reviewing and editing)

Data Availability: Data and the R script, Rmarkdown file, Supplementary information and materials from the empirical studies are available at on the Open Science Framework at https://osf.io/nztuk/. The study was preregistered at https://osf.io/ch7n9.

Competing interests: The authors declare no competing interests.

Funding: This research was supported by an Australian Research Council grant (FT190100708) to the last author and an Australian Government Research Training Program Scholarship to the first author.

Ethics approval and consent to participate: All procedures were approved by the University of Western Australia’s Human Research Ethics Office (Ethics ID: 2019/RA/4/20/6423) and complied with all relevant guidelines and the Declaration of Helsinki. All participants provided informed consent prior to participating.

Consent for publication: Informed consent included consent for publication of aggregated data.

Ecker, U. K. H. et al. The psychological drivers of misinformation belief and its resistance to correction. Nat. Rev. Psychol. 1, 13–29 (2022)
Ferreira Caceres, M. M. et al. The impact of misinformation on the COVID-19 pandemic. AIMS Public Heal. 9, 262–277 (2022)
Kozyreva, A., Lewandowsky, S. & Hertwig, R. Citizens versus the internet: Confronting digital challenges with cognitive tools. Psychol. Sci. Public Interes. 21, 103–156 (2020)
Kozyreva, A., Smillie, L. & Lewandowsky, S. Incorporating psychological science into policy making. Eur. Psychol. 28, 206–224 (2023)
Kozyreva, A. et al. Toolbox of interventions against online misinformation and manipulation. PsyArXiv Prepr. 1–24 (2022) doi:10.31234/OSF.IO/X8EJT
Walter, N. & Murphy, S. T. How to unring the bell: A meta-analytic approach to correction of misinformation. Commun. Monogr. 85, 423–441 (2018)
Chen, S., Xiao, L. & Kumar, A. Spread of misinformation on social media: What contributes to it and how to combat it. Comput. Human Behav. 141, 107643 (2023)
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G. & Rand, D. G. Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychol. Sci. 31, 770–780 (2020)
Roozenbeek, J., van der Linden, S., Goldberg, B., Rathje, S. & Lewandowsky, S. Psychological inoculation improves resilience against misinformation on social media. Sci. Adv. 8, (2022)
Lin, Y., Osman, M. & Ashcroft, R. Nudge: Concept, effectiveness, and ethics. Basic Appl. Soc. Psych. 39, 293–306 (2017)
Pennycook, G. & Rand, D. G. Nudging social media toward accuracy. Ann. Am. Acad. Pol. Soc. Sci. 700, 152–164 (2022)
Pennycook, G. & Rand, D. G. Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation. Nat. Commun. 2022 131 13, 1–12 (2022)
Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595 (2021)
Epstein, Z. et al. Developing an accuracy-prompt toolkit to reduce COVID-19 misinformation online. Harvard Kennedy Sch. Misinformation Rev. (2021) doi:10.37016/mr-2020-71
Prike, T., Butler, L. & Ecker, U. Source-credibility information and social norms improve truth discernment and reduce engagement with misinformation online. (2023) doi:10.31234/OSF.IO/DHX6F
Roozenbeek, J., Freeman, A. L. J. & van der Linden, S. How accurate are accuracy-nudge interventions? A preregistered direct replication of Pennycook et al. (2020). Psychol. Sci. 32, 1169–1178 (2021)
Altay, S., Berriche, M. & Acerbi, A. Misinformation on misinformation: Conceptual and methodological challenges. Soc. Media + Soc. 9, 205630512211504 (2023)
Zollo, F. & Quattrociocchi, W. Misinformation spreading on Facebook. in Complex Spreading Phenomena in Social Systems (eds. Lehmann, S. & Ahn, Y.) 177–196 (2018). doi:10.1007/978-3-319-77332-2_10
Chater, N. & Loewenstein, G. The i-frame and the s-frame: How focusing on individual-level solutions has led behavioral public policy astray. Behav. Brain Sci. 1–60 (2022) doi:10.1017/S0140525X22002023
Tay, L. Q., Lewandowsky, S., Hurlstone, M. J., Kurz, T. & Ecker, U. K. H. A focus shift in the evaluation of misinformation interventions. Harvard Kennedy Sch. Misinformation Rev. (2023) doi:10.37016/mr-2020-124
Bhardwaj, V., Martel, C. & Rand, D. G. Examining accuracy-prompt efficacy in combination with using colored borders to differentiate news and social content online. Harvard Kennedy Sch. Misinformation Rev. (2023) doi:10.37016/mr-2020-113
Butler, L. H. et al. The (Mis)Information Game: A social media simulator. Behav. Res. Methods (2023) doi:10.3758/s13428-023-02153-x
Guay, B., Berinsky, A. J., Pennycook, G. & Rand, D. How to think about whether misinformation interventions work. Nat. Hum. Behav. 7, 1231–1233 (2023)
Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175–191 (2007)
Chen, X., Pennycook, G. & Rand, D. What makes news sharable on social media? J. Quant. Descr. Digit. Media 3, (2023)
Globig, L. K., Holtz, N. & Sharot, T. Changing the incentive structure of social media platforms to halt the spread of misinformation. Elife 12, (2023)
Christensen, R. H. B. Cumulative link models for ordinal regression with the R Package ordinal. Unsubmitted Manuscr. 1–40 (2018)
Wickham, H. Data Analysis. 189–201 (2016) doi:10.1007/978-3-319-24277-4_9
Langsrud, Ø. ANOVA for unbalanced data: Use Type II instead of Type III sums of squares. Stat. Comput. 13, (2003)
Herve, M. RVAideMemoire: Testing and plotting procedures for biostatistics. at https://cran.r-project.org/package=RVAideMemoire (2022)
Lenth, R. V. et al. emmeans: Estimated Marginal Means, aka Least-Squares Means. at (2023)
Swart, J., Peters, C. & Broersma, M. Navigating cross-media news use. Journal. Stud. 18, 1343–1362 (2017)
Mathews, N., Bélair-Gagnon, V. & Lewis, S. C. News is “toxic”: Exploring the non-sharing of news online. New Media Soc. 146144482211272 (2022) doi:10.1177/14614448221127212
Borges do Nascimento, I. J. et al. Infodemics and health misinformation: A systematic review of reviews. Bull. World Health Organ. 100, 544–561 (2022)

No competing interests reported.

Supplementcomplete.pdf

Download PDF

Editorial decision: Revision requested
19 Jan, 2024
Reviews received at journal
16 Jan, 2024
Reviewers agreed at journal
28 Dec, 2023
Reviewers invited by journal
28 Dec, 2023
Editor assigned by journal
28 Dec, 2023
Editor invited by journal
12 Dec, 2023
Submission checks completed at journal
12 Dec, 2023
First submitted to journal
10 Dec, 2023

You are reading this latest preprint version

Nudge-Based Misinformation Interventions are Effective in Information Environments with Low Misinformation Prevalence

Status:

Version 1

Abstract

Figures

Full Text

Method

Participants

Materials

Nudge Intervention

Posts

Procedure

Results

Impact of Nudge Intervention on Engagement Behavior

Impact of Misinformation Proportion on Engagement Behavior

Does the Effectiveness of the Nudge Intervention Decay Over Time?

Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1