Null effects of news exposure: A causal test of the (un)desirable effects of a ‘news vacation’ and ‘news binging’

This preregistered project examines the general belief that news has a beneﬁcial impact on society. We test news exposure effects on desirable outcomes, i.e., political knowledge and participation, and detrimental outcomes, i.e., attitude and affective polarization, negative system perceptions, and worsened individual well-being. We rely on two complementary over-time experiments that combine participants’ survey self-reports and their behavioral browsing data: one that incentivized participants taking a ‘news vacation’ for a week (N = 797; 30M visits) in the US, the other of ‘news binging’ for two weeks (N = 828; 17M visits) in Poland. Across both experiments, we demonstrate that reducing or increasing news exposure has little – if any – impact on the positive or negative outcomes tested. These robust null effects emerge irrespective of participants’ prior levels of news consumption and whether prior news diet was like-minded, and regardless of compliance levels. We argue that these ﬁndings reﬂect the reality of limited news exposure in the real world, with news exposure comprising roughly 3.5% of citizens’ online information diet.

Our study offers several key advantages. For one, we maxi-53 mize ecological validity by embedding the treatments in partic-54 ipants' real life rather than in a controlled and isolated context. 55 As importantly, we move beyond reliance on self-reports by 56 analyzing participants' online browsing data comprising over 57 47 million visits, collected via our open-source tool Web His-58 torian (see SI A.1). We use these behavioral data to measure 59 compliance, establish floor and ceiling effects, and examine 60 heterogeneity in treatment effects by prior levels of news con-61 sumption and also the congeniality thereof. Toward this end, 62 we create a comprehensive list of news domains in both coun-63 tries (4,683 in the US, 301 in Poland, of which 944 and 212, 64 respectively, were visited by our final samples). We match 702 for 133 of the visited news domains in Poland (see SI A.2). We 68 guard against several threats to our conclusions (e.g., attrition 69 bias) and account for differential levels of compliance measured 70 using both self-reported and online behavioral data. 71 We find a robust pattern of near-zero effects. Neither taking 72 a week-long news vacation nor increasing news consumption 73 for two weeks influenced the tested outcomes, beneficial (e.g., 74 political engagement) or not (e.g., polarization, attribution 75 of malevolence to out-party). These null effects emerged 76 consistently regardless of one's prior levels of news exposure, 77 the extent to which one's news diet was like-minded, and  The 'news vacation' experiment was embedded in Wave 3 99 of the US panel survey. The 872 respondents who took part 100 in Wave 3 were invited to take part in the experiment and 803 101 agreed to participate (92%) and were randomly assigned to an 102 experimental or control condition (probability of assignment 103 1 Web Historian is a web browser plug-in that accesses respondents' browser history stored on their computers, displays it to them using visualizations (e.g. network graph of websites visited, word cloud of used search terms, searchable table of browser history), and allows them to submit it to researchers following an extensive informed consent. SI A.1 contains more details and shows screenshots of the interactive informed consent process to treatment: 60%). The treatment participants (N = 457) were incentivized to stop following the news for one week, every 28 sites they visited. Centrist sites were most popular 139 (53%), and visits to like-minded domains accounted for 28% of 140 news visits with ideological classification (or mere 0.80% of all 141 browsing!). These descriptives offer one crucial insight: News 142 is only a small drop in an ocean of online content, and so it 143 is questionable whether changing this small part of people's 144 information diet will make any difference. We return to this 145 finding in the discussion.  Fig. 2 shows the 161 results for the 'No News' experiment in yellow and those for 162 the 'More News' experiment in grey. 3 . SI C disaggregates 163 the results for the individual items of the composite outcome 164 variables. The dataset and the replication code are available 165 on Harvard Dataverse and Github. 166 We first address the beneficial outcomes: political knowl-167 edge and participation (we do not have pre-measurements for 168 these variables). Unlike hypothesized, participants who con-169 sumed more news were not any more knowledgeable (Facet ii) -170 or felt they were (Facet i) -than the control. In addition, those 171 in the No News condition were not any less knowledgeable 172 than those in the control, nor did they feel as such. Similar 173 null effects from 'news vacation' and 'news binging' emerge 174 for participants' engagement in a range of civic and political 175 activities, from signing a petition to protesting (Facet iii). 176 We turn to negative outcomes, testing if news exposure 177 increases attitude polarization (i.e., attitude importance and 178 strength on five salient issues per country) and affective po-179 larization (i.e., hostility toward out-ideologues, out-partisans, 180 and citizens with opposite policy beliefs, each measured in 181 three ways), see SI A.3. Using multiple measures ensures that 182 the detected patterns are not due to any specific measurement 183 alone. Again, the over-time treatment -whether decreasing 184 or increasing news use -had no significant effects on attitude 185 (Facets iv for importance and v for strength) and affective 186 (Facets vi to viii) polarization. Treatment effects do not sur-187 pass the 2% mark independently of which indicator and which 188 political out-group we examine.

189
Adding to this null pattern, news exposure had near-zero 190 effects on three negative system perceptions: whether people 191 think the out-party wants to harm the country (Attribution of 192 malevolence, Facet ix)), oppose politicians crossing the aisle 193 and reaching compromise (No support for compromise, (Facet 194 x), and perceive the political climate as polarized (Perceived 195 polarization, Facet xi), even though -theoretically -media's 196 focus on negativity (16), conflict, horse-race, and in-your face 197 debates (17-19) was expected to worsen these outcomes.

198
Lastly, we predicted that news exposure would reduce indi-199 vidual well-being. Studies find links between news consump-200 tion and stress, anxiety, fatigue, or sleep loss (Jan 31, 2018, 201 click here), especially when news is personally relevant (30), 202 and negative effects of hard news exposure on one's mental 203 well-being (31). These emotional responses may trigger un-204 healthy behaviors to alleviate the stress. Yet, our causal tests 205 find no significant news effects on emotional well-being (e.g., 206 anxiety, anger, among other emotions) and physical well-being 207 (e.g., consuming alcohol, getting into arguments, wanting to 208 hit someone) during the treatment period (Facets xii and xi). 209 Robustness checks. To ascertain that these (near-zero) effects 210 are robust, we test whether our treatment has different effects 211 depending on one's prior news diet. For instance, some partic-212   consumption and its congeniality, news effects did not depend 258 on individual typical news diet. That is, the decrease in news 259 use was not less impactful for the avid news consumers or the 260 increase in news use did not affect those rarely exposed to news.

261
Similarly, changes in one's news diet did not depend for the 262 respondents who more clearly complied with the treatments. behavioral traces, in that no such work can claim representa-273 tiveness. 4 Also, it is possible that people selectively shape 274 what content they opt out of in a way that preserve their exist-275 ing attitudes. In other words, participants may have complied 276 in volume but adjusted sources or content in ways that buffers 277 any potential change. In a similar vein, our findings cannot 278 speak to the content seen by the participants. We attributed 279 the potential negative effects to various biases in journalistic 280 routines, yet the news our participants typically see may not 281 be about negativity, conflict, or polarization. News content 282 aside, the robust null pattern is noteworthy.

283
These results counter the popular narrative that news media 284 contribute to healthy citizenry and our expectations that 285 they should have a range of adverse effects. Nevertheless, we 286 argue that these effects portray the reality of (very limited) 287 effects of news exposure in the real world more accurately. 288 Past work cannot speak to actual exposure in naturalistic 289 settings, where people can select from unlimited content and 290 where politics accounts for a small fraction of citizens' online 291 activities. In our data, spanning six months of individual 292 web browsing, visits to news websites comprised 3.54% of the 293 overall browsing. This is normatively problematic, as citizens 294 should stay informed about politics. At the same time, this 295 finding puts into perspective news media effects altogether. 296 Because news content is nearly unnoticeable in the context 297 of overall information and communication ecology of most 298 individuals, as we show, its effects are also very limited. This 299 evidence aligns with the vast literature on minimal media 300 impact.

301
Naturally, news media are important. They keep other 302 powers in check by investigating and publicizing the truth, 303 offer information, and bind citizens together around shared 304 events or concerns. Furthermore, news media may still play a 305 paramount role in the development of political attitudes during 306 political socialization (33) and have cumulative effects on 307 people's perceptions of (political) reality (34), long-term effects 308 that we cannot capture. This project, however, the first to rely 309 on incentivized over-time designs using both self-reported and 310 online behavioral indicators in naturalistic settings and across 311 countries, suggests that the contributions of news media may 312 be more limited than typically hoped or assumed.

314
See SI Appendix for a detailed description of all materials and meth-315 ods used within this study as well as additional robustness checks, 316 extended discussion of the used classifiers as well as alternative 317 classifications. The data and the code will be made available 318 upon publication on GitHub and on Harvard Dataverse.    175-194 (1998 Web Historian tool on the participants' local computers. Participants went through the nine steps process pictured 390 below but the data visualized was their own web browsing data and differed for each participant. The web browsing 391 data pictured are example data that are not from a participant. outlet has a more liberal audience and higher scores indicate a more conservative audience. Using these scores, we 398 categorized the domains as either liberal, centrist, or conservative, such that liberal news sites were those with an 399 ideological score of -.20 or lower, conservative sites included those with scores of .09 or higher, and news sites with 400 scores between -.19 and .08 were categorized as centrist.

401
These categorizations were based on natural cut points in the data that made intuitive sense and had face validity. 402 Because our dataset is public, these categories can be reassigned. Appendix Figure A.2 visualizes the ideology ranking 403 of all sites that were visited at least 5 times by our participants and had ideology scores.

A.2.2. Poland.
In the absence of parallel scores for Poland, we rely on a technique that uses follower patterns of news 405 media accounts on Twitter. We start with the list of news organizations compiled earlier, but only consider those (1) 406 that have a visit frequency in our data of above the median, or are national outlets even though less visited (2) and 407 that have a Twitter account. This leaves us with 153 domains in Poland.

408
Our scaling approach builds on the "mediascores" model by Eady and Linder (click here), which is based on the 409 assumption of homophily: users on social media, conceived as a one-dimensional ideological space, are more likely to 410 share news from news media accounts close to them. Instead of using sharing behavior, we use following behavior, 411 thus assuming that users are more likely to follow news organizations close to them.

412
To build the bipartite graph that indicates whether any user follows any media account, we obtain the list of 413 Twitter followers of all media organizations. To avoid an overly sparse graph, we exclude organizations with less 414 than 250 followers. To better estimate ideology scores for small media accounts, we first look at accounts with less 415 than 30,000 followers, and get all followers who follow at least 10 of them. We take all of these users into account. 416 Then, for the media accounts with more than 30,000 followers, we only pull a random sample of 300 followers. For 417 validation purposes, we also add parliament members as followers to the graph, again excluding those with less than 418 250 followers.

Fig. A.2. Ideology classification American domains
Notes. The horizontal axis signals the ideology estimates. Lower and negative scores indicate a more liberal and higher scores more conservative share of audience responding to the outlet. The size of the points represents the logged number of visits in our data. The news domains list was compiled by manually coding the domains listed in Alexa's Top 1000, the 1000 most browsed domains in our own data and the 1000 most shared domains by politicians on Twitter. Only sites that were visited five times or more are displayed in this figure. The full table containing the raw scores is available in the data folder in the replication repository as ' Figure A.2[A.3] -Data.csv'.

Fig. A.3. Ideology classification Polish domains
Notes. The horizontal axis signals our own ideology estimates based on Twitter linking patterns. Lower and negative scores indicate a more liberal and higher scores more conservative share of audience responding to the outlet. The size of the points represents the logged number of visits in our data. The news domains list was compiled by manually coding the domains listed in Alexa's Top 1000, the 1000 most browsed domains in our own data and the 1000 most shared domains by politicians on Twitter. Only sites that were visited ten times or more are displayed in this figure. The full table containing the raw scores is available in the data folder in the replication repository as ' Figure A. * in Poland out-partisans were defined as (a) people who hold opposite stances on the government (b) people who support the largest party on the opposite side of the spectrum, and (c) people who support the party respondents feel farthest from. ** a separate item for each of the four issues listed above. Notes. We find a correspondence between the rank of the news sites included in our study and the rank of the news sites using site rankings from Alexa (we note that Alexa has no visit statistics for Google News and Yahoo News). As this table shows, the top browsed news sites reported by Alexa are also among the top browsed sites in our samples.   Notes. Table shows balance statistics between treatment and control for those accepting participation, and separately for those responding to the post-experiment survey. Columns labelled "Sign." show significance tests (p-value of a chi-squared test) for differences between treatment and control, or differences between those accepting and responding to post-survey within a condition. Notes. Table shows balance statistics between treatment and control for those accepting participation, and separately for those responding to the post-experiment survey. Columns labelled "Sign." show significance tests (p-value of a chi-squared test) for differences between treatment and control, or differences between those accepting and responding to post-survey within a condition.

Fig. B.1. Power
Notes. Figure is based on G*Power analyses for linear fixed effects analysis (F test-family) with 2 as the denominator of the degrees of freedom for the main effect models and 3 for the moderation effect models. Effect sizes below 0.2 are considered very small effect sizes, effect sizes between 0.2 and 0.5 small, and between 0.5 and 0.8 medium. Notes. Table displays the summary statistics of the variables the way they appear in the analyses. The dependent variables in the analyses were rescaled to range between 0 and 100. The exposure variables were constructed by dividing the number of visits by the number of days an individual logged onto the computer (active days). These variables were log-transformed and subsequently rescaled to range between 0 and 100. Notes. The chord diagrams visualize the overtime change in the values in the dependent variables. Arcs within the same category or colors (for example from grey '0-20' to grey '0-20) indicate the percentage of respondents who reported no change between the pre-and post-measurement. Arcs between categories (for example from grey '0-20' to dark blue '20-40') denote the percentage of respondents who reported a change from one category to the other between the two timepoints. The 'messier' the diagram is, the larger the overtime variability.

Fig. B.3. Overtime variability dependent variables Poland
Notes. The chord diagrams visualize the overtime change in the values in the dependent variables. Arcs within the same category or colors (for example from grey '0-20' to grey '0-20) indicate the percentage of respondents who reported no change between the pre-and post-measurement. Arcs between categories (for example from grey '0-20' to dark blue '20-40') denote the percentage of respondents who reported a change from one category to the other between the two timepoints. The 'messier' the diagram is, the larger the overtime variability.

Fig. B.4. Variable distributions United States
Notes. All variables were rescaled to range between 0 and 100, as is the case in the analyses, to make the distributions easier to read.

Fig. B.5. Variable distributions Poland
Notes. All variables were rescaled to range between 0 and 100, as is the case in the analyses, to make the distributions easier to read. Notes. The horizontal bars indicate a 95% confidence interval surrounding the point estimate. Model 1 is based on a fixed effects model. Models 2 and 3 are based on a random effects model with a cross-level interaction between the news exposure variables and the experimental manipulation. All exposure measures were log-transformed to account for the skewed distribution. The dependent variables were rescaled between 0 and 100 so that the coefficients denote the percentual change in the dependent variable as the result of one unit increase in the independent variable. The table containing the raw scores is available in the output/tables folder in the replication repository as ' Figure  C.1[C.10] -Data.xlsx'.

Fig. C.2. Disaggregated analyses attitude polarization
Notes. The horizontal bars indicate a 95% confidence interval surrounding the point estimate. Model 1 is based on a fixed effects model. Models 2 and 3 are based on a random effects model with a cross-level interaction between the news exposure variables and the experimental manipulation. All exposure measures were log-transformed to account for the skewed distribution. The dependent variables were rescaled between 0 and 100 so that the coefficients denote the percentual change in the dependent variable as the result of one unit increase in the independent variable. The table containing the raw scores is available in the output/tables folder in the replication repository as ' Figure  C.1[C.10] -Data.xlsx'.

Fig. C.3. Disaggregated analyses affective polarization
Notes. The horizontal bars indicate a 95% confidence interval surrounding the point estimate. Model 1 is based on a fixed effects model. Models 2 and 3 are based on a random effects model with a cross-level interaction between the news exposure variables and the experimental manipulation. All exposure measures were log-transformed to account for the skewed distribution. The dependent variables were rescaled between 0 and 100 so that the coefficients denote the percentual change in the dependent variable as the result of one unit increase in the independent variable. The table containing the raw scores is available in the output/tables folder in the replication repository as ' Figure  C.1[C.10] -Data.xlsx'.

Fig. C.4. Disaggregated analyses negative system perceptions
Notes. The horizontal bars indicate a 95% confidence interval surrounding the point estimate. Model 1 is based on a fixed effects model. Models 2 and 3 are based on a random effects model with a cross-level interaction between the news exposure variables and the experimental manipulation. All exposure measures were log-transformed to account for the skewed distribution. The dependent variables were rescaled between 0 and 100 so that the coefficients denote the percentual change in the dependent variable as the result of one unit increase in the independent variable. The table containing the raw scores is available in the output/tables folder in the replication repository as ' Figure  C.1[C.10] -Data.xlsx'.

Fig. C.5. Disaggregated analyses general well-being
Notes. The horizontal bars indicate a 95% confidence interval surrounding the point estimate. Model 1 is based on a fixed effects model. Models 2 and 3 are based on a random effects model with a cross-level interaction between the news exposure variables and the experimental manipulation. All exposure measures were log-transformed to account for the skewed distribution. The dependent variables were rescaled between 0 and 100 so that the coefficients denote the percentual change in the dependent variable as the result of one unit increase in the independent variable. The table containing the raw scores is available in the output/tables folder in the replication repository as ' Figure  C.1[C.10] -Data.xlsx'.