The study was pre-registered prior to data collection (https://osf.io/nwk7x).
Methods
Participants
All participants were recruited through the Department of Psychology participant pool at the University of Illinois, Urbana-Champaign. All participants gave their informed consent prior to the inclusion in the study. The work was approved by the Office for Protection of Research Subjects (OPRS) at the University of Illinois. All procedures were performed in accordance with the University of Illinois IRB and the Declaration of Helsinki. 221 native English speakers (mean age: 19; 144 female, 77 male) completed all tasks online.
Materials
Experimental sentences: 120 sentences all 12 words each were split in 2 lists in a Latin-square design and presented in a whole sentence self-paced reading fashion. Sentences included 20 simple active sentences, 20 passive main clauses, and 80 sentences containing relative clauses taken from Traxler et al. 38 (40 subject relative clauses (SRC) and 40 object relative clauses (ORC)). Sentences were followed by comprehension questions (See Appendix for a complete list of sentences and questions). Items were pseudorandomized such that no two items of the same kind followed each other. The number of SRC and ORC was doubled relative to simple and passive sentences because SRC and ORC sentences were constructed in pairs (e.g., The lawyer that the banker… and The banker that the lawyer...) so participants saw only half of the experimental items. The order of the lexical items for the sentential arguments was counterbalanced. For example, if a participant viewed the SRC with the head noun lawyer then they would have viewed the ORC with the head noun banker. The question phrasing and the order of answer options was counterbalanced as well such that there was an equal number of “yes no” and “no yes” displays and an equal number of “yes” and “no” responses.
1. Simple sentence: I went to the store and bought milk, eggs, and green beans.
Did I go to the library?
Yes No
2. Passive main clause: Yesterday morning, the nurse was helped by the cowboy in ripped jeans
Did the cowboy help the nurse last week?
No Yes
3. Subject relative clause: The lawyer that irritated the banker retrieved the paperwork from the office.
Did the lawyer irritate the banker?
Yes No
4. Object relative clause:The lawyer that the banker irritated retrieved the paperwork form the office
Did the banker irritate the lawyer?
No Yes
Text exposure surveys
Assessing reading experience is not straightforward. Adults tend to exaggerate reading habits so indirect measures such as Author Recognition Tasks (ART) circumvent social desirability and yield better estimates (39).
Author Recognition Test: We used an updated version of the Acheson, et al. (2008)15 by Moore and Gordon (2015)40. See Supplemental materials for the full survey. This task asks participants to choose real authors from a list of names (60 real, 60 foil authors). Participants received 1 point for a real author and 1 point was subtracted if participants chose a foil name.
Reading Enjoyment Survey: To build converging measures of reading experience, we adapted a survey that measured reading enjoyment in children (41) to survey reading enjoyment in adults. This survey consists of 10 statements that asked participants to either agree or disagree on a 1 through 7 Likert scale about various reading attitudes to assess participants attitudes and intrinsic motivation associated with reading (e.g., I enjoy reading; I enjoy receiving books as gifts). See Supplemental Materials for the full survey. A composite score was computed as the average of all 10 responses. For the questions that probed negative attitudes the scores were flipped to keep positive values at the higher end of the scale.
One motivation for this survey was to obtain a convergent measure of text exposure to complement the ART. Positive attitudes and intrinsic motivation are associated with reading frequency (42) so we hypothesized assessing attitudes towards reading may allow us to indirectly assess reading experience. A second motivation arises from challenges associated with collecting data online. Even software that locks participants’ screens and prevents them from surfing the internet while performing a task cannot prevent participants from using their phones to look up whether the author is real or not. Assessing reading attitudes may minimize opportunities for participants to “cheat” even if social desirability may become a greater concern.
Vocabulary Test (43): Participants were asked to choose a synonym for 40 real English words out of 4 possible variants for each word. Given a suspicious number of perfect or very high scores, it was evident that participants used their phones or other devices to look up correct synonyms for this test. We do not discuss the results further because we believe the results are not reliable.
Demographics Survey: In-house developed survey that collected basic information pertaining to participants age, gender, SES, reading ability, disability, or dyslexia diagnoses. See Supplemental materials for full version of the survey)
Procedure
Participants were given a link after they chose to participate in our experiment through the SONA participant pool administration software. First, they gave consent to participate. Then they were directed to the website that displayed the sentence reading portion of the experiment followed by ART, Vocabulary, Reading Experience survey and basic demographics questionnaire. The experiment was implemented in Ibex farm online software (44). Ibex farm uses JavaScript and html forms to collect participant responses and response times on the participant’s own computer and uploads participants responses to the server only after participants hit “Finish” button on last page of the experiment. Such approach minimized the response time delays for the reading time measures.
Data Exclusion Criteria
Participants who learned English after 5 years of age (N= 40) or reported a history of reading difficulties (N=23) were excluded from the analyses.
A substantial challenge with online data collection is that it tends to be noisier than data collected in the lab. We developed a pipeline to remove trials and participants that did not likely reflect true reading processes (e.g., “button mashing,” careless clicking, or multitasking during study participation). For the response times, a two-step process was used: first, reading times faster than 1500 ms and slower then 138,000 ms (2.3 minutes – highest cut off time used traditionally for one-word-at-a-time self-paced reading studies of 11,500 ms was multiplied by 12 corresponding to 12 words in all our sentences) were excluded (3,513 data points removed out of the total 21,826 points). Second, the individual reading times were trimmed to cut off 2.5 standard deviations above and below the individual conditional mean (additional 853 data points reducing the dataset to 17,460 total data points). Based on these exclusions the total number of participants was reduced from 280 to 241. Additionally, participants were excluded if previous trimming left less than 50% of items for each of the 4 sentence types (additional 20 participants). Finally, based on our prior observations of individuals completing these tests in person in the lab setting it takes about 5 minutes maximum to complete the ART test. As a result, we excluded participants who took longer than 300,000 ms (5 min) to complete the test (N=10 participants). As a result of all the exclusion criteria the final dataset contained 211 participants – 64% of the participants who took part in our experiment (343 total participants). These exclusion rates are consistent with other online studies that find 45% to 53% of participants/trials are removed (45).
Statistical variables, contrasts, and model fitting considerations
Text exposure surveys (ART, and RE) and sentence type (active, passive, SRC, ORC) were used to predict sentence reading times and comprehension question accuracy. Reading times were analyzed using linear mixed-effects (LME) models, and accuracy results were analyzed with generalized LMM (GLMM) models using the lme4 package (Version1.1-13; 46) in R (Version 3.2.0; 47). Three orthogonal contrasts were specified through dummy coding to compare relative clause versus main clause sentences, followed by active versus passive sentences and SRC versus ORC sentences. This coding scheme was preregistered. Three contrasts were defined: relative clauses: SRC “1” and ORC “1” versus main clauses: Active “-1” and Passive “-1” (MC vs RC); ORC versus SRC: SRC “-1” , ORC “1”, Active “0” and Passive “0”; passive versus active sentences: SRC “0” , ORC “0”, Active “-1” and Passive “1”. Additionally, as an exploratory analyses after viewing the results, we used treatment contrast where each of the three complex structures were compared to active sentences that served as a baseline. Measure of text exposure (ART and RE scores) were centered and scaled.
LME models were fitted to untransformed and log-transformed reading times (See Supplemental materials, Tables 1 & 2 for model results). The results of the transformed and untransformed times were remarkably similar, so we report the untransformed models to facilitate interpretation. We note any significant differences in the pattern of results. The random structure was determined following Barr et al (2013)48 maximal fit approach. LME models were fit by restricted maximal likelihood with the Satterthwaite’s method; generalized LME models were fit by maximum likelihood with Laplace approximation. P-values were obtained through summary function of the lmerTest package (49). The final models for reading times have random slopes for items and participants. The final accuracy models have random slopes for items only due to convergence failure. The exploratory model for accuracy with both ART and RE did not converge with random structure, as a result we fitted this model with regular regression (lm instead of glmer).
Response time and accuracy plots in Figure 2 were inspired by van Langen’s open visualizations (2020)50
Results
Assessments of Text Exposure
The two assessments of text exposure, the Author Recognition Task (Mean: 13.13, SD = 6.21, Range = -1-30) and Reading Enjoyment Survey (Mean: 4.27, SD = 1.55, Range = 1.2-7) were only moderately correlated; readers with more positive attitudes recognized more real authors (r=.33, p<.001). In subsequent analyses, we probe whether the measures each capture variance in our sentence processing measures.
Reading Analyses
Whole-sentence reading time analyses were limited to items on which the participant correctly answered the comprehension question. As expected, participants read the simple sentences faster and more accurately (Figure 2 & Table 2) than rarer or more syntactically complex sentences. However, relative rankings across the four sentence types for speed and accuracy were not the same. ORC sentences took the longest time to read, followed by SRC, passive, and simple sentences. However, accuracy was the lowest for passive sentences, followed by ORC, then SRC and simple sentences.
Sentence Type
|
N Correct
|
Reading Times Mean (milliseconds)
|
SE
|
N Overall
|
Accuracy
|
SE
|
Active
|
3981
|
4009
|
59.71
|
4074
|
.98
|
.00
|
Passive
|
3153
|
5592
|
98.59
|
4176
|
.76
|
.01
|
SRC
|
2233
|
6121
|
117.99
|
2520
|
.89
|
.01
|
ORC
|
2043
|
7099
|
126.89
|
2549
|
.80
|
.01
|
Table 2. Participants means, sentence counts and standard errors (SE) for reading times and accuracy rates by sentence type
Effects of Text Exposure on Reading Time and Accuracy
Effects of Author Recognition Test
To test our key hypothesis, we investigated the how reading experience affected both overall reading times and question accuracy, and how reading times interacted with sentence type. For visualization purposes, Figure 3 shows the relationship between ART (top row) and RE survey (bottom row) and reading times (first column) and comprehension question accuracy (second column).
Models predicting reading times revealed main effects and an interaction between sentence types and ART score (Table 3, Model 1). All participants read relative clause sentences slower than main clause sentences (main effect of MC vs RC), passives slower than active sentences (main effect of Active vs Passive), and ORCs slower than SRCs (SRC vs ORC). However, the interactions show that participants with higher ART scores read relative clauses (versus main clauses) and passive sentences (versus the active sentences) faster than participants with lower ART scores (ART interaction with MC vs RC and Active vs Passive). Participants with more text exposure showed smaller differences in reading times for the easier and harder sentences. Log-transformed data revealed a similar pattern of results, except that the main effect for the ART was not reliable (see Supplemental materials, Exhibit A Table A, Model 1).
When we used our exploratory treatment contrast in the same model (Table 3, Model 2) we get very similar results. All three sentence types were read slower than the active sentences. There was no main effect of the ART but it interacted with all three comparisons. Log-transformed models revealed identical results (Supplemental Material, Exhibit A, Table A, Model 2).
Across all models, we see a clear effect of text exposure on reading times. We see some evidence that participants who had higher ART score read faster overall, and converging evidence that participants who had higher ART scores were especially faster to read passive, SRC and ORC sentences, the sentence types more frequent in written language, than participants who had lower ART scores.
Measure
|
Contrast
|
b
|
SE
|
t/z
|
Model 1: Preregistered dummy coding scheme
RT ~ MC_RC*ART+A_P*ART+SRC_ORC*ART+(1|Item)+(1|Participant)
|
RT in ms
|
Intercept
|
5765.46
|
187.87
|
30.688***
|
|
MC vs RC
|
969.14
|
101.29
|
9.57***
|
|
Active vs Passive
|
821.65
|
149.54
|
5.49***
|
|
SRC vs ORC
|
587.01
|
134.84
|
4.35***
|
|
ART
|
-502.95
|
163.77
|
-3.07**
|
|
MC vs RC: ART
|
-136.07
|
43.52
|
-3.13**
|
|
Active vs Passive: ART
|
-164.09
|
52.94
|
-3.10**
|
|
SRC vs ORC: ART
|
-36.79
|
68.74
|
-0.54
|
Model 2: Condition treatment contrasts
RT~ A_P*ART+A_SRC*ART+A_ORC*ART+ (1|Item)+(1|Participant)
|
RT in ms
|
Intercept
|
3974.67
|
261.88
|
15.18***
|
|
Active vs Passive
|
1643.29
|
299.09
|
5.49***
|
|
Active vs SRC
|
2172.91
|
282.69
|
7.69***
|
|
Active vs ORC
|
3346.93
|
285.07
|
11.74***
|
|
ART
|
-202.78
|
172.75
|
-1.17
|
|
Active vs Passive: ART
|
-328.18
|
105.88
|
-3.10**
|
|
Active vs SRC: ART
|
-473.02
|
118.21
|
-4.00***
|
|
Active vs ORC: ART
|
-399.44
|
122.24
|
-3.27**
|
Model 3: Preregistered dummy coding scheme
RT ~ MC_RC*ART+A_P*ART+SRC_ORC*ART+(1|Item)+(1|Participant)
|
Accuracy
|
Intercept
|
3.20
|
.17
|
18.85***
|
|
MC vs RC
|
-1.17
|
.16
|
-7.24***
|
|
Active vs Passive
|
-2.85
|
.39
|
-7.36***
|
|
SRC vs ORC
|
-.44
|
.15
|
-3.01**
|
|
ART
|
.23
|
.07
|
3.11**
|
|
MC vs RC: ART
|
.06
|
.06
|
1.12
|
|
Active vs Passive: ART
|
.11
|
.11
|
.95
|
|
SRC vs ORC: ART
|
.02
|
.04
|
.44
|
Model 4: Condition treatment contrasts
RT~ A_P*ART+A_SRC*ART+A_ORC*ART+ (1|Item)+(1|Participant)
|
Accuracy
|
Intercept
|
4.36
|
.29
|
14.97***
|
|
Active vs Passive
|
-2.85
|
.39
|
-7.36***
|
|
Active vs SRC
|
-1.89
|
.36
|
-5.32***
|
|
Active vs ORC
|
-2.77
|
.35
|
-7.88***
|
|
ART
|
.30
|
.12
|
2.55*
|
|
Active vs Passive: ART
|
.11
|
.11
|
.95
|
|
Active vs SRC: ART
|
.11
|
.13
|
.87
|
|
Active vs ORC: ART
|
.15
|
.12
|
1.24
|
Table 3. LME Models predicting the untransformed reading times with condition and ART for pre-registered and exploratory (treatment) contrasts. Note: ^p<.1; *p<.05; **p<.01; ***p<.001
Generalized LME predicting comprehension question accuracy with ART scores and sentence type revealed only main effects of sentence type and text exposure for both pre-registered and exploratory contrasts (Table 3 Models 3 and 4). Participants were overall less accurate on relative clauses and passive sentences than active sentences and participants with more text exposure were overall more accurate on all sentence types.
Effects of Reading Enjoyment Survey
Reading Enjoyment scores showed an identical pattern of effects on reading times as did ART scores for both our preregistered and exploratory model contrasts (Table 4, Model 1 & 2). Log-transformed data revealed identical pattern of results to the raw data with two exceptions: only passive versus active sentences contrast (not the main versus relative clause) yielded significant interaction with the Reading Enjoyment score and both models with pre-registered and exploratory contrasts revealed the main effect of Reading Enjoyment (Supplemental Materials, Exhibit B, Table B, Models 1 & 2). Effects of Reading Enjoyment scores on comprehension question accuracy were also nearly identical to those of the ART, when using the preregistered contrasts (Table 4 Model 3). However, the same model with the exploratory treatment contrasts converged only with random slopes for items, not participants. Given the potential problems with model fit, we additionally include results from a linear regression model (Table 4, Model 5). We observe no main effect of Reading Enjoyment but see significant interactions between Reading Enjoyment and sentence types such that participants who reported higher degrees of reading enjoyment tend to be more accurate in comprehending all three types of rare or complex sentences relative to the simple sentences than participants who enjoy reading less.
Measure
|
Contrast
|
b
|
SE
|
t/z
|
Model 1: Preregistered dummy coding scheme
RT~MC_RC*RE+A_P*RE+SRC_ORC*RE+(1|Item)+(1|Participant)
|
RT in ms
|
Intercept
|
5759.64
|
188.64
|
30.53***
|
|
MC vs. RC
|
972.47
|
102.43
|
9.49***
|
|
Active vs Passive
|
820.13
|
151.40
|
5.42***
|
|
SRC vs. ORC
|
590.06
|
136.15
|
4.33***
|
|
RE
|
-486.69
|
165.03
|
-2.95**
|
|
MC vs. RC: RE
|
-119.61
|
43.27
|
-2.76**
|
|
Active vs Passive: RE
|
-124.09
|
53.42
|
-2.32*
|
|
SRC vs. ORC: RE
|
-57.89
|
67.54
|
-0.86
|
Model 2: Condition treatment contrasts
RT~ A_P*RE+A_SRC*RE+A_ORC*RE + (1|Item)+(1|Participant)
|
RT in ms
|
Intercept
|
3967.04
|
264.12
|
15.02***
|
|
Active vs Passive
|
1640.26
|
302.81
|
5.42***
|
|
Active vs SRC
|
2175.01
|
285.90
|
7.61***
|
|
Active vs ORC
|
3355.13
|
288.30
|
11.64***
|
|
RE
|
-242.99
|
174.09
|
-1.40
|
|
Active vs Passive: RE
|
-248.17
|
106.84
|
-2.32*
|
|
Active vs SRC: RE
|
-421.19
|
117.74
|
-3.58***
|
|
Active vs ORC: RE
|
-305.41
|
120.80
|
-2.53*
|
Model 3: Preregistered dummy coding scheme
Accuracy~MC_RC*RE+A_P*RE+SRC_ORC*RE+(1|Item)+(1|Participant)
|
Accuracy
|
Intercept
|
3.19
|
.17
|
18.84***
|
|
MC vs. RC
|
-1.15
|
.16
|
-7.13***
|
|
Active vs Passive
|
-2.82
|
.39
|
-7.29***
|
|
SRC vs. ORC
|
-.44
|
.15
|
-3.00**
|
|
RE
|
.24
|
.07
|
3.32***
|
|
MC vs. RC: RE
|
.03
|
.05
|
.56
|
|
Active vs Passive: RE
|
.02
|
.11
|
.15
|
|
SRC vs. ORC: RE
|
.01
|
.04
|
.03
|
Model 4: Condition treatment contrasts
Accuracy~ A_P*RE+A_SRC*RE+A_ORC*RE + (1|Item)
|
Accuracy
|
Intercept
|
4.14
|
.28
|
15.03***
|
|
Active vs Passive
|
-2.72
|
.37
|
-7.36***
|
|
Active vs SRC
|
-1.84
|
.34
|
-5.39***
|
|
Active vs ORC
|
-2.68
|
.34
|
-7.97***
|
|
RE
|
.22
|
.10
|
2.10*
|
|
Active vs Passive: RE
|
.02
|
.05
|
.86
|
|
Active vs SRC: RE
|
.02
|
.05
|
.40
|
|
Active vs ORC: RE
|
.01
|
.04
|
.14
|
Model 5 (lm): Accuracy~ A_P*RE+A_SRC*RE+A_ORC*RE
|
Accuracy
|
Intercept
|
.98
|
.01
|
184.95***
|
|
Active vs Passive
|
-.22
|
.01
|
-29.91***
|
|
Active vs SRC
|
-.09
|
.01
|
-10.66***
|
|
Active vs ORC
|
-.18
|
.01
|
-20.67***
|
|
RE
|
.01
|
.01
|
.93
|
|
Active vs Passive: RE
|
.03
|
.01
|
3.50***
|
|
Active vs SRC: RE
|
.02
|
.01
|
2.29*
|
|
Active vs ORC: RE
|
.03
|
.01
|
3.79***
|
Table 4. LME Models for the untransformed reading times, accuracy rates and RE results for pre-registered and exploratory (treatment) contrasts Note: ^p<.1; *p<.05; **p<.01; ***p<.001
Variance accounted for by ART and Reading Enjoyment scores
In exploratory follow-up analyses, ART and Reading Enjoyment scores were put in the same model with the pre-registered contrasts and exploratory treatment contrasts to investigate whether the two measures of text exposure accounted for similar or different sources of variance in reading times and comprehension accuracy. Full models are presented in supplementary materials (Supplemental materials, Exhibit C, Table C, Models 1-4). Despite the relatively low correlation between the two measures of text exposure (r = 0.33) and that each independently predicted reading times, we found no evidence that the inclusion of both ART and Reading Enjoyment in a model predicting reading times improved fit over including only a single predictor. Though we find some evidence that ART and RE may account for non-overlapping variance in comprehension question accuracy, given potential issues with model convergence and data sparsity, we cannot strongly draw this conclusion. We delegate it to future studies to investigate the sources of similarities and differences between ART and Reading Enjoyment further.