As noted earlier, external researchers who were not the program designers[1] conducted two studies, the results of which we present below. Students’ reading skills in English were assessed at the beginning (pretest) and at the end (posttest) of the school year, using the Group Reading Assessment and Diagnostic Evaluation (GRADE)[2] as a standardized measure. An arithmetic measure, the Wide Range Achievement Test (WRAT), was also included to control for the Hawthorne effect[3].
The experiment took place over two full school years (2004-2005 and 2006-2007). In 2004-2005, the explicit ADOR-IIR instructional programs described below were tested with 117 children from 6 classes in 2 schools. In 2006-2007, the ADOR-IIR programs were tested again with 87 pupils from 4 classes in the same 2 schools as the previous year. Two teachers participated in both experiments. A total of 204 pupils experienced the ADOR-IIR programs and made up the experimental group. The control group consisted of 276 children from 13 classes in 7 schools who received the standard reading instruction described below. Reading instruction in both the experimental and control groups was provided by teachers who had between 1 and 37 years of teaching experience. There were no significant differences in the average number of years of teaching experience between teachers in the experimental and control groups. School’s district administrators selected which schools would take part, and teachers agreed to participate in the study after an information session.
The ADOR-IIR experimental group: Implementing the ADOR program in regular classes with all first-grade pupils
Intended for all first-grade students, the ADOR program aims to develop children’s reading skills in their first 80 to 100 days of school. Both the ADOR program (the English-language version) and the DAL program (the French-language version) draw on evidence of effective instruction in learning to read (National Reading Panel, 2000), Explicit Teaching of reading comprehension (Boyer, 1993), and Direct Instruction (Carnine, Silbert & Kameenui, 1997), as well as on research on resilience and optimism (Seligman, 1995), Kirschner’s (2002) work on cognitive load, and research on behavioural interventions and token-saving systems that promote effort and academic achievement (Little & Akin-Little, 2003). The ADOR, like its French-language counterpart, is clearly an explicit and direct instructional program that emphasizes the development of reasoning while reading, skills for retrieving information in a read text, systematic acquisition of decoding, and the increase in fluency, including prosody.
ADOR is also consistent with current evidence recommending the development of vocabulary and general world knowledge, which are recognized as crucial to reading comprehension (Oakhill, Cain, & Elbro, 2015; Stuart & Stainthorp, 2016; Willingham, 2017).
At the beginning of the learning process, the ADOR program includes doing daily phonological-skill activities, learning of phonemes systematically, decoding simple words, understanding sentences from short texts read to or with pupils, reasoning while reading texts or sentences read to or with pupils, and developing vocabulary. After 14 days of instruction, some of the most common letter combinations in English are systematically introduced (e.g., ay, ea, oo and sh), letter activities are greatly reduced, and phonological activities and some decoding activities are omitted altogether. Reasoning, comprehension, and vocabulary activities are retained throughout the year. An oral reading fluency activity is incorporated into daily activities when pupils[4] can read aloud at an exact-rate of 12 to 19 words per minute.
Brief and frequent formative assessments are provided throughout the program, among other objectives, to identify the weakest pupils and to adjust instruction and activities to the group’s average performance. The two to five pupils in the class with the weakest reading skills then receive additional ad hoc support from the teacher, called ad hoc remediation. This help is provided individually or in small groups for 5-10 minutes per day. After approximately 10 days, the pupils are re-assessed. The results of the second evaluation determine whether the intervention was effective and whether the student will continue to receive ad hoc support from the teacher.
At the beginning of the school year, ADOR classroom teachers attended in a four-day training session given by the school board’s trained coaches. After the training session, the coaches demonstrated specific pedagogical activities and classroom management techniques in regular classes (with the pupils and their teacher) and observed the extent to which the teachers were applying the content from the training sessions and the coach’s demonstrations. Observation grids are used and teachers received feedback. Each teacher was observed at least 20 times per year (the duration of an observation/demonstration, including feedback, varied from 40 to 75 minutes).
The ADOR-IIR experimental group: Implementing the IIR program with pupils with the weakest reading skills in the ADOR program in first-grade
The ADOR program in grade 1 is accompanied by the IIR remedial program with the goal to recover pupils with the weakest reading skills. As mentioned above, the program is a translation and adaptation of the DIR reading support program (see Boyer, 2010, for a more detailed description), an existing French-language curriculum in Québec. The IIR program for Grade 1 rolls out in the winter or spring, and targets the first graders with the weakest reading skills who also participate in the ADOR program in their regular classroom. These children are identified by measuring their oral reading of a text that they have not read before. Another measurement, of comprehension and reasoning on silent reading, is made using one or more small texts that the children have never read before. Then, the school’s 8-12 lowest-performing first graders are selected, regardless of other criteria (e.g., hyperactive, dyslexic, behaviour problems, language at home, or having cooperative parents).
Like its French-language counterpart, IIR is an explicit and direct reading instruction program. This remedial program rolls out over an 8-10-week period, with 2 consecutive hours of intervention per day. In all, the IIR program must include a minimum of 76 hours of intervention. The basis of the IIR program, which is applied by the school’s special-aid teachers, includes activities from ADOR program, with some special features for students who are at-risk or have learning disabilities. In addition, certain empirical data specific to remedial intervention are incorporated into the DIR program (Boyer, 2010). The content of the intervention is similar to the activities from the regular classroom, but the pace is faster, the instructional materials are different, and some of the activities and specific classroom-management techniques are adapted. The program aims to reduce the gap between IRR students and the other pupils in the regular classroom.
During the school year, IRR special-aid teachers participated in eight days of training led by the school board’s trained coaches. The coaches followed up in the classroom (with the pupils and their teacher) to demonstrate instructional activities and classroom‑management techniques and observe the teachers’ level of application. Observation grids were used and special-aid teachers received feedback. Each remedial teacher was observed at least 12 times per year (an observation/demonstration including feedback lasted a full day).
The control group: Use of the Québec Ministère de l’éducation curricula in regular classes with all first-grade pupils
The control-group classes followed the Québec Ministère de l'éducation’s regular curricula (2001), which was monitored by the school board’s pedagogical services staff. The Whole Language and constructivist approaches strongly influenced the pedagogical services staff’s discourse and recommendations for how the curriculum must be applied. In short, the recommended approach to reading instruction is to read large texts (from big books or children’s literature) every day to the group, to guide reading with questions, to have pupils globalize specific words, to emphasize comprehension and sense-making in reading, and to verbalize words using anticipation, often based on the illustration, title, context, and other words in the sentence or text. Ideally, these tasks are done as part of a “meaningful and authentic” educational project based on student interests that leads to some form of production (writing, drawing, etc.). Writing and reading are closely integrated into these projects and informal formative assessments (student interviews, work samples, etc.) are preferred. The creation of a portfolio (student work samples) reflecting the student’s development in reading and writing is encouraged. Decoding (phonics) is not prohibited, but it is recommended that it be done briefly, only when needed and in response to difficulties in reading.
Monitoring of the application of this pedagogy is ensured through various trainings offered by the school board’s pedagogical services staff whom are responsive to teachers’ requests and usually provide training that can lead to follow-up in the classroom. However, we do not know how many training sessions and follow-ups have been offered and what kind of feedback was used.
The control group: Special-aid teachers using a variety of remedial interventions with the weakest pupils in reading skills in first-grade regular classes which uses the Québec Ministère de l’éducation curriculum
All classes in the control group received remedial instruction. Usually, home-teachers select pupils with weak skills to receive remedial services from special-aid teachers, and the process is monitored by the principal. Delivery of special-aid services vary. There is the “pull-out,” students are taken out of class 1-3 times a week for 20-40 minutes sessions, depending on the child’s difficulties and the special-aid teacher’s availability. Students may be seen individually or in small groups. The remedial progam is similar to regular class content and activities, sometimes using the same materials and sometimes using different materials. Some special-aid teachers may use also more play-based activities in their interventions. Others provide activities that complement the learning in the regular classroom. Some schools ask special-aid teachers to work exclusively or partially right in the regular classroom. Usually the special-aid teacher working in the regular classroom, helps the teacher by working with the weaker pupils. Sometimes the home-teacher and special-aid teacher work as a team, sharing tasks and pupils. Supervision of remedial services is provided by the school board and its educational consultants. The latter usually respond to the requests of special-aid teachers by providing materials, training, or support. We do not know how much training and follow-up was provided to special-aid teachers or what type of feedback they used.
Standardized measurement instruments used by Savage (2005), and Savage and Deault (2007)
The Group Reading Assessment and Diagnostic Evaluation (GRADE) is a standardized reading and listening-comprehension test has Canadian norms. GRADE has proven to be reliable and valid for measuring reading and listening skills. It can be administered to an entire class of young children.
Pretests
The GRADE’s Word reading, Word meaning, and Listening comprehension subtests were administered during the pretest between September 15 and October 15, in both 2004 and 2006. These subtests were administered in the classroom to all the children at once for both the experimental and control groups.
In the Word Reading subtest, children are asked to check one of the four words per item. The word is first read aloud by an examiner, then used in a sentence, and finally repeated. In the Word Meaning subtest, children are asked to read a word and select one of the four pictures that best depicts its meaning. In the Listening Comprehension subtest, children are asked to listen to a short sentence or excerpt and to select one of the four pictures that best depicts the meaning of the text they just heard.
Posttests
Posttests were administered between April 15 and the end of June 2005 and in May 2007. The Listening Comprehension, Word Reading, and Word Meaning subtests were again used. Two additional GRADE subtests, Sentence Comprehension and Reading Comprehension of text passages, were added. For the Sentence Reading Comprehension subtest, students are asked to read a sentence silently and select the word (out of four) that best matches the sentence. For the Passage Comprehension subtest, students silently read a short passage of three to four sentences. They then read a question and select an answer among four choices.
Hawthorne effect
To control for the Hawthorne effect, the arithmetic subtest of the Wide Range Achievement Test (WRAT) was administered. Its first 10 items were presented to pupils at both the pretest and the posttest.
Additional measures
Additional qualitative assessments were also conducted. The researchers interviewed teachers and observed the classrooms, using the Atmosphere, Instruction, Management, Student Engagement (AIMS) test developed by Pressley et al. 2001. In addition, pupils completed the Classroom Environment Scale (CES; Fraser 1986), which measures their perceptions of the classroom. The results of these additional measures were correlated with student progress in reading.
Hypotheses[5]
Since ADOR and IIR are partly based on Explicit and Direct Instruction frameworks and build on the evidence for learning to read, we postulated the following:
1. The ADOR-IIR group is expected to outperform the control group on direct measures of reading learning in first-grade, particularly in Word reading, Sentence reading comprehension, and Passage reading comprehension.
Since learning to read and reading itself can be influenced by oral language development (Lervåg, Hulme, & Melby-Lervåg, 2018):
2. It is expected that the ADOR-IIR group will outperform the control group on measures of Word meaning and Listening comprehension.
Since reading is not very solicited in first-grade math and even less so in arithmetic:
3. The ADOR-IIR group should perform similarly to the control group on the arithmetic measure.
[1] The designer of ADOR and IIR, Christian Boyer, is one of the authors of this text.
[2] The Group Reading Assessment and Diagnostic Evaluation (GRADE™) is a diagnostic reading test that determines the achieved level of reading and listening skills of pupils in grades K-12.
[3] The Hawthorne effect refers to the effect of simply participating in an experiment, which tends to lead to greater motivation among subjects in the experimental group compared to subjects in the control group, which may affect the results of the experimental group.
[4] Exact rate is the number of words students read correctly in one minute without help when reading a text out loud that they have never seen before.
[5] These are the authors’ (Bissonnette, Boyer, & Morneau-Guérin) hypotheses and not those of Savage and Deault (2007).