The effects of observing, imagining, and imitating the instructor’s depictive gestures on learning from instructional videos

doi:10.21203/rs.3.rs-2852608/v1

Download PDF

Research Article

The effects of observing, imagining, and imitating the instructor’s depictive gestures on learning from instructional videos

https://doi.org/10.21203/rs.3.rs-2852608/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

The instructor’s depictive gestures in instructional videos are beneficial for learning, and learning strategies are crucial to make full use of them. This study adopted a within-subjects design to investigate the effects of learning strategies (i.e., observing, imagining, and imitating the instructor’s depictive gestures) on learning from instructional videos (i.e., learning performance, cognitive load, learning efficiency, learning satisfaction, and attention allocation). The repeated-measures ANOVAs revealed that when using imitation strategy, students showed the best learning performance, learning efficiency, and satisfaction. The Friedman tests results showed that when using imagination and imitation strategies, students’ attention was distracted by the instructor. Additionally, spatial ability played a moderation role in transfer performance. The results lead to a strong recommendation for educational practice when using instructional videos: (1) when watching videos with the instructor’s depictive gestures, students are encouraged to use imitation strategy in preference to imagination strategy, and passive observation is not recommended; and (2) low-spatial-ability students are encouraged to use active learning strategies (i.e., imagining and imitating the instructor’s depictive gestures) to improve transfer performance.

depictive gesture

learning strategy

imagination

imitation

spatial ability

Learning from instructional videos is a prevalent learning activity in formal and informal environments. When instructors express difficult learning materials, they often use depictive gestures accompanied by speech to provide semantic information. For instance, when the instructor talks about “the separation of chromosomes during mitosis”, it is indicated by the two hands moving away. Depictive gestures are the mental modeling of movement and perception, which can provide more visual input information and semantic content (Mayer & Fiorella, 2022).

However, previous studies did not gain a consistent conclusion on the effects of depictive gestures, which may be attributed to the use of learning strategies (Willems & Hagoort, 2007; Son et al., 2018; Ianì et al., 2018; Pi et al., 2019). There are three common learning strategies used in learning from videos: observation, imagination, and imitation. Observation is a passive learning strategy, and imagination and imitation are two active learning strategies related to gestures.

As a passive learning strategy, observing gestures did not always guarantee the effectiveness of learning (Pi et al., 2019; Brucker et al., 2022). Additionally, studies have shown that compared to the observation strategy, the two active strategies of imagination and imitation take advantage of the instructor’s depictive gestures (Baills et al., 2018; García-Gámez et al., 2021; Leopold et al., 2019). Imagination is an effective generative learning strategy, helping students construct mental imagery, but increasing the cognitive load (Leahy & Sweller, 2004; Fiorella & Mayer, 2015; Leopold et al., 2019). Unlike imagination, the imitation strategy not only is an effective generative learning strategy but externalizes mental representations and frees up cognitive resources, thus reducing the cognitive load (James & Swain, 2011; Baills et al., 2018; García-Gámez et al., 2021). So far, to our knowledge, no empirical study compared the effects of three strategies on video learning including an instructor’s depictive gestures.

In addition, past studies have proved that spatial ability moderated the effectiveness of the design of learning materials and the application of learning strategies (Lee, 2007; Brucker et al., 2015, 2022). The ability-as-compensator hypothesis held that the design of learning materials and instruction that was beneficial to low-spatial-ability students may not be helpful for high-spatial-ability students (Höffler, 2010). However, to our knowledge, little was known about the moderation role of spatial ability on the effects of observing, imagining, and imitating an instructor’s depictive gestures in instructional videos .

1.1. The effects of depictive gestures on learning from instructional videos

Gestures are identified as spontaneous and meaningful hand movements that are usually accompanied by speech, and convey information to the audience, which are widely used by the instructors during instruction (McNeill, 1992; Goldin-Meadow, 2014). Depictive gestures is a kind of typical gesture, which describes various concrete or abstract aspects of semantic content through the shape or movement track of gestures. Depictive gestures express the spatial properties of objects or related events, and evoke the mental imagery of the shape in students’ minds in a literal or metaphorical way (e.g., crossing the hands left and right to express the concept of “air convection”; Alibali et al., 2013; Goldin-Meadow, 2014; Pi et al., 2019, 2021). Also, depictive gestures often rely on the accompanying speech to complement meaningful information for the instructor’s interpretation, which will lead to a deep understanding of the concept (Willems & Hagoort, 2007).

The potential benefits of the instructors’ depictive gestures in instructional videos were supported by the dual channels principle of multimedia learning (Congdon et al., 2017; Pi et al., 2019; Mayer & Fiorella, 2022). The dual channels principle holds that students process information through visual and auditory channels, which work independently but with limited capacity. When students learn new materials, their limited working memory capacity will affect their acquisition of new knowledge (Mayer & Fiorella, 2022). Therefore, if the information is presented in both visual and auditory channels, it will expand their effective working memory capacity, thereby reducing their cognitive load, and thus facilitating the acquisition of new knowledge (Paas & Van Merriënboer, 1994; Mayer & Fiorella, 2022).

Numerous empirical studies have confirmed the benefits of the instructor’s depictive gestures on learning from instructional videos (Son et al., 2018; Ianì et al., 2018; Pi et al., 2019). For example, Son et al. (2018) used mathematical concepts “variance and mean” to explore the effects of depictive gestures on undergraduate students’ conceptual understanding. They found that watching instructional videos with the instructor’s depictive gestures was beneficial for students to understand complex concepts, which was probably because the instructor’s depictive gestures help students pay attention to and remember key knowledge points, so as to understand the learning materials better. A similar result was also shown in an eye movement study by Pi et al. (2019). To be specific, when learning “the adjustment of the curve in Photoshop”, students who viewed the instructional video with the instructor’s depictive gestures gained better transfer performance and paid more attention to the gestures than those who viewed the video without gestures. The above evidence supported the embodied cognition theory and the dual channels principle of multimedia learning (Wilson, 2002; Mayer & Fiorella, 2022).

However, the instructor’s depictive gestures did not always facilitate students’ learning from videos (Pi et al., 2019). For instance, although Pi et al. (2019) observed the positive effects of depictive gestures on students’ transfer, they failed to find the benefits of the instructor’s depictive gestures on retention performance and attention allocation. A study conducted by Brucker et al. (2022) with the theme of “fish movement patterns” also found no significant differences in classification performance between students watching videos with and without the instructor’s depictive gestures.

Taken together, the instructor’s depictive gestures had wide influences on students’ learning performance, cognitive load, and attention allocation, but the results were controversial. The above inconsistent results about the effects of instructor’s depictive gestures on learning from videos might be attributed to the use of learning strategies.

1.2. Comparisons of observation, imagination, and imitation strategies

Besides the design of instructional videos, the effectiveness of learning strategies in video learning also attracted much attention (Fiorella & Mayer, 2016). Three learning strategies related to depictive gestures were examined: observation, imagination, and imitation of gestures (Brucker et al., 2015; Pi et al., 2019; García-Gámez et al., 2021).

Observational learning was defined as learning by viewing the actions of others (Bandura, 1986), which has been proved to improve learning performance and learning experience (Marcus et al., 2013; Castro-Alonso et al., 2014; Fiorella & Mayer, 2016). The instructor’s depictive gestures can provide social cues that help students to understand the learning material by associating the instructor’s gestures with relevant cognitive processing. The positive effects of observing gestures were usually attributed to the activation of the mirror neuron system. The brain regions activated when the students observed the instructor’s gestures in the video were the same as those activated when the students performed the same gestures themselves (Marcus et al., 2013; Holle et al., 2008; Brucker et al., 2015). However, as a passive learning strategy, observation did no always work, for example, Pi et al. (2019) found no beneficial effect of observing an instructor’s depictive gestures on knowledge retention.

Imagination strategy indicated that students process knowledge by imagining in the mind, which was one of the eight generative learning strategies (Fiorella & Mayer, 2015; Leopold et al., 2019). According to generative learning theory, learning by imagining activated the processes of selection (selecting relevant components of learning material to include in mental imagery), organization (arranging the components in mental imagery spatially), and integration (connecting the learning content with prior knowledge). Learning by imagining encouraged students to mentally visualize knowledge concepts or processes in working memory, forming meaningful constructs that are transferred and stored in long-term memory, which improved learning performance and learning efficiency (Lin et al., 2016; Cooper et al., 2001; Yang et al., 2019; García-Gámez et al., 2021). For instance, García-Gámez et al. (2021) tested the impact of depictive gestures on vocabulary learning in a foreign language, and they found that compared to viewing videos without gestures, those who observed and imagined the instructor’s depictive gestures gained a higher percentage of recall. However, imagination strategy may harm learning when the integration of information may exceed the limit of cognitive capacity and result in a higher cognitive load (Leahy & Sweller, 2004).

Imitation involved the observation and execution of other’s actions (Tellier, 2008; Mainieri et al., 2013; García-Gámez et al., 2021), which can not only provide semantic information but also activate students’ motor and perception systems, thus affecting the cognitive process (James & Swain, 2011; García-Gámez et al., 2021). Embodied cognition theory provided explanations for the positive effects of imitating the instructors’ depictive gestures (Wilson, 2002). According to the embodied cognition theory, an individual’s thinking and cognition are derived from the sensory and motor experience of the body, and there is a strong connection between the physical experience and the mental state. From this perspective, when observing and executing gestures, learners may have a joint understanding of the mental imagery with the instructor in their sensory and motor systems (Alibali & Nathan 2012; Ping et al. 2014). The beneficial effects were also associated with the mirror neuron system (Rumiati & Bekkering, 2003; Mainieri et al., 2013). Specifically, when students are asked to imitate the instructor’s gestures, they will have a shared mental state. To some extent, imitation also involved the process of observation and imagination, and only by generating information in the mind can the observation be externalized.

Moreover, studies have shown that externalized knowledge can unload information accumulated in cognitive capacity, avoid cognitive overload, and improve learning performance (Baills et al., 2018; Yang et al., 2021b; García-Gámez et al., 2021). For example, Baills et al. (2018) explored the influence of depictive gestures on students’ learning of Chinese tones. They found that compared to the passive observation of gestures, students who watched videos using the imitation strategy could identify Chinese words and tones better. The study by García-Gámez et al. (2021) also proved that imitating the instructor’s depictive gestures was more effective than imagining gestures during learning vocabulary in a foreign language.

To sum up, the existing studies have shown that observation, imagination, and imitation of gestures had different influences on learning. To illustrate, the observation strategy worked by activating the mirror nervous system; the imagination strategy helped students build mental images, but may increase cognitive load; while the imitation strategy can externalize mental representations and free up cognitive resources. However, studies on the effectiveness of observing, imagining, and imitating gestures mainly concentrated on the learning performance and cognitive load of one or two strategies, and little was known about the differences in attention allocation among the three strategies.

1.3. The moderation role of spatial ability

Spatial ability was regarded as an individual ability at the processing of dynamic visualizations as well as observing and making gestures (Brucker et al., 2015, 2022). Specifically, spatial ability had to do with an individual’s ability to perceive the attributes (e.g., location, shape, etc.) of objects, form a mental representation of the attributes, and manipulate the representations mentally (Höffler, 2010; Brucker et al., 2022). Previous studies have shown that spatial ability moderated the effectiveness of material design (Lee, 2007; Brucker et al., 2015). For example, Brucker et al. (2015) found that low-spatial-ability students performed better in the well-designed (i.e., viewing corresponding depictive gestures) group than in the poor-designed (i.e., viewing non-corresponding depictive gestures) group when they learned from the videos to classify the fish movement patterns; while for high-spatial-ability students, no difference was found. The results were consistent with the ability-as-compensator hypothesis, that was, low-spatial-ability students might need well-designed visualization to achieve good learning performance, but high-spatial-ability students did not (Höffler, 2010). The study conducted by Brucker et al. (2022) further proved that students’ spatial ability may influence the effectiveness of learning strategies in a fish movement patterns classification task. The results found that low-spatial-ability students who learned without gestures performed better than those who observed and made gestures; but for low-spatial-ability students, making gestures and learning without gestures did not differ. In contrast to their hypothesis, making gestures did not improve learning performance, which may be due to the ambiguous self-gesturing instruction.

In sum, the effectiveness of the learning strategies differed for students with different spatial ability levels. Therefore, we assume that spatial ability will moderate the effects of observing, imagining, and imitating the instructor’s depictive gestures on learning from instructional videos.

1.4. The present study

Taken together, studies have shown that an instructor’s depictive gestures in video lectures influences students learning performance, cognitive load, and attention allocation (Son et al., 2018; Ianì et al., 2018). However depictive gestures does not always facilitate learning from videos (Pi et al., 2019). It should be noted that the effects of the instructor’s depictive gestures are varied with students’ learning strategies of observation, imagination, and imitation (Marcus et al., 2013; Fiorella & Mayer, 2016; Cooper et al., 2001; Yang et al., 2019; García-Gámez et al., 2021). To our knowledge, there has been no study of the horizontal comparison of the learning effects of observing, imagining, and imitating the instructor’s depictive gestures. Besides, the moderating role of spatial ability on the three strategies has not been comprehensively understood.

This study aimed to compare the effects of observing, imagining, and imitating the instructor’s depictive gestures on learning performance, cognitive load, learning efficiency, learning satisfaction, and attention allocation. In addition, we expected to verify the moderation role of spatial ability. The following hypotheses were proposed.

H1: Students will show the best learning performance (retention and transfer) when they used the imitation strategy to learn a video (Imitating gestures condition), followed by using imagination strategy (Imagining gestures condition), next to using observation strategy (Observing gestures condition), finally by passively viewing a video without gestures (No gestures condition).

H2: Students in the Imitating gestures condition will report the least cognitive load, followed by Observing gestures condition, next to No gestures condition, finally by Imagining gestures condition.

H3: Students in the Imitating gestures condition will show the highest learning efficiency, followed by Imagining gestures condition, next to Observing gestures condition, finally by No gestures condition.

H4: Students in the Imitating gestures condition will report the highest learning satisfaction, followed by Imagining gestures condition, next to Observing gestures condition, finally by No gestures condition.

H5: Students in the Imitating gestures condition will spend the least percentage fixation duration on slides and the most on the instructor, followed by Imagining gestures condition, next to Observing gestures condition, finally by No gestures condition.

H6: Spatial ability will moderate the effect of learning strategy on learning performance (retention and transfer), cognitive load, learning efficiency, learning satisfaction, and attention allocation. In other words, there will be an interaction effect between spatial ability and learning strategy.

2.1. Participants and design

A priori power analysis performed with G*power indicated that the required size was 30 (effect size f = 0.25, α = 0.05, power = 0.90, number of measurements = 4; Faul et al., 2007). Thus, thirty undergraduate and graduate students from a Chinese university were recruited to participate in the study. Participants ranged in age from 18 to 23 (M_age = 20.60, SD_age = 1.52; 6 males, and 24 females), and they reported a normal or corrected-to-normal vision and hearing. Each participant provided informed consent. They received 50 CNY and some gifts (e.g., tissues) as a reward at the end of the experiment. The study protocol was approved by the local ethics committee.

The experiment used a within-subjects design and counterbalanced Latin square design, in which each participant experienced four conditions: (a) No gestures condition, (b) Observing gestures condition, (c) Imagining gestures condition, and (d) Imitating gestures condition.

2.2. Instructional videos

Four videos were developed for each condition. The themes “The stars in the universe”, “Marine topography”, “Geological structure”, and “The earth’s atmosphere” were used for the No gestures condition, Observing gestures condition, Imagining gestures condition, and Imitating gestures condition respectively.

In the No gestures condition, the instructor in the video stood still and had no gesture at all (Fig. 1a), and participants were asked to passively view the video. In the Observing gestures condition, participants were asked to passively view the video, in which the instructor used a total of 30 depictive gestures to express the meaning of related words or phrases. For example, when the instructor said “the cross-section of the trench is asymmetrical in the shape of a V”, she uses two palms together and open up to make the shape of a “V”, with one palm tilted up and the other palm tilted down to make the “V” asymmetrical (Fig. 1b). In the Imagining gestures condition, the instructor in the video used a total of 30 depictive gestures, and participants were asked to imagine the instructor’s depictive gestures in their minds at the same time(Fig. 1c). In the Imitating gestures condition, the instructor in the video used a total of 30 depictive gestures, meanwhile, participants were asked to imitate the instructor’s depictive gestures (Fig. 1d). When asked whether they had imagined and imitated the instructor’s depictive gestures in the Imagination and Imitation conditions, all participants reported that they had done as the requirements. Therefore, the experimental manipulations were valid.

The length of the videos ranged from 236 s to 240 s (M = 238.00, SD = 1.83). Twelve students from different majors rated the difficulty and knowledge type of the four videos. The results of repeated-measures ANOVA showed no significant difference in the difficulty, F(3, 33) = 0.31, p = .819, η_p² = 0.03. Additionally, all raters reported that the knowledge type of four videos was homogeneous, and the order of four videos will not influence learning.

2.3. Measurements

2.3.1. Spatial ability test

Spatial ability was measured by a mental rotation test consisting of 24 items (Wong et al., 2018). Each item included five cubical figures, in which the left-most figure was the target figure, two of the four figures to the right of the target figure were rotated versions of the target, and the other two figures could not be rotated into congruence with the target figure (see Fig. 2). Participants were given six minutes to mark the two figures that could be rotated to match the target figure. If both rotated figures were identified, one point was scored. Scores were the total of correct answers, with a maximum of 24 points. Participants were classified as having low (n = 16) or high (n = 14) spatial ability according to their scores above or below the mean score (M = 12.60, SD = 5.49). And the further independent samples t-test on mental rotation test showed significant difference between them, t(29) = 8.52, p < .001, Cohen’s d = 0.84.

2.3.2. Prior knowledge test

A prior knowledge test containing four sub-tests was developed by the instructor in the video to assess participants’ general knowledge of the four themes. Each sub-test included two multiple-choice items (one point for the correct answer) and fill-in-blank items with three blanks (one point for each blank). The maximum score for each sub-test was five points, and the internal consistencies of the four sub-tests were moderate (Cronbach’s α = 0.71, 0.69, 0.67, and 0.53, respectively). The results of repeated-measures analysis of variance (ANOVA) showed that there was no significant difference in the scores of prior knowledge tests across the four conditions, F(3, 87) = 1.04, p = .381, η_p² = 0.03.

2.3.3. Learning performance tests

Four learning performance tests were developed for four themes, and each test included a retention test and a transfer test. The types of items for the four sub-tests were different, but their full scores were the same (50 points). The internal consistencies of the four sub-tests were moderate (Cronbach’s α = 0.62, 0.63, 0.69, and 0.62, respectively).

The retention test was used to measure participants’ memory of the information covered in the video, which included multiple-choice items (two points for the correct answer), and fill-in-blank items (one point for each blank). And the full score was 36 points. The transfer test was used to measure participants’ understanding and application of the knowledge covered in the video, with a maximum score of 14 points. It included multiple-choice items (two points for the correct answer), fill-in-blank items (one point for each blank), and open-ended items (five points for each item).

2.3.4. Cognitive load scale

The mental effort was usually used to measure the cognitive load by one 9-point Likert item (“How much mental effort did you invest for the learning just now?”, Paas & van Merriënboer, 1994; Hefter & Berthold, 2020), rated from 1 (extremely little) to 9 (extremely much).

2.3.5. Learning efficiency

Learning efficiency refers to the degree to which students can improve their learning performance with the same amount of mental effort (van Gog & Paas, 2008; Yang et al., 2021b). Scores on mental effort and learning performance test (the sum scores of retention and transfer) were used to calculate the learning efficiency, with the formula: Learning efficiency = (Z_{learning performance} − Z_{mental effort}) / √2 (van Gog & Paas, 2008; Moning & Roelle, 2021). A value less than zero indicates the learning efficiency decreased, and a value greater than zero means the learning efficiency improved.

2.3.6. Learning satisfaction scale

Learning satisfaction indicates the degree to which students are satisfied with whether learning activities meet their needs, which includes the degree of satisfaction with teaching, learning content, interaction, and learning environment (Yang et al., 2021a). The learning satisfaction scale containing 17 items was rated from 1 (strongly disagree) to 5 (strongly agree). The internal consistencies of the scale for four tests were high (Cronbach’s α = 0.91, 0.93, 0.92, and 0.93, respectively).

2.4. Eye movement data collection and analysis

Eyelink 1000 eye tracker (SR Research Ltd., Canada) was used to record participants’ eye movements with a sampling rate of 1000 Hz. The instructional videos were presented on a 17-inch monitor with a resolution of 1024 × 768 pixels and a refresh rate of 75 Hz. The participants sat 65 cm away from the monitor.

To measure participants’ attention allocation, two areas of interest (AoIs) were created (slides area and instructor area; Fig. 3). Then, the percentage of fixation duration on each AoI was adopted (i.e., the fixation duration on each area / the total fixation time × 100%), which refers to the cognitive process of attending to an AoI (Yang et al., 2021a).

2.5. Procedure

The experiment was conducted in an eye movement laboratory. At the beginning of the experiment, the participants learned about the experimental procedure, signed the informed consent form, and completed the demographic questionnaire (gender, age, and major; 5 min). Second, they filled out the mental rotation test, and prior knowledge test (20 min). Third, the participants were asked to learn the video under one of the four conditions (No gestures, Observing gestures, Imagining gestures, and Imitating gestures). It should be emphasized that before viewing the video, students were told in detail how to use the corresponding strategy in the learning process. Then they completed the cognitive load scale, learning satisfaction scale, and learning performance test. After that, they had a 5-min rest and then moved on to the next condition. The procedure was show in Fig. 4.

Firstly, the scores on the learning performance test, cognitive load, learning efficiency, and learning satisfaction across the four conditions all showed a normal distribution and homogeneity of variance (ps > .05), so a series of repeated-measures ANOVAs were conducted to test H1, H2, H3, and H4. Secondly, the results of all eye movement indicators did not show a normal distribution (ps < .001), and thus Friedman non-parametric tests were conducted on them to test H5. Finally, a repeated-measures ANOVA was conducted to examine the moderation effect of spatial ability on learning performance, cognitive load, learning efficiency, learning satisfaction, and attention allocation (H6), with the condition as a within-subject factor and spatial ability (low vs. high) as a between-subject factor. The descriptive statistics for all dependent variables are shown in Table 1.

Table 1

Means (M) and standard deviations (SD) for all dependent variables.
Dependent variables	No gestures	Observing gestures	Imagining gestures	Imitating gestures
Prior knowledge test (0–5)	0.87 (0.82)	1.00 (1.02)	1.10 (1.03)	1.27 (1.20)
Retention (0–36)	16.90 (0.85)	18.87 (1.00)	20.50 (1.12)	23.10 (0.74)
Transfer (0–14)	5.37 (3.25)	6.53 (2.20)	6.90 (2.80)	9.35 (2.86)
Total (0–50)	22.27 (6.89)	25.40 (6.46)	27.40 (7.94)	32.45 (7.94)
Cognitive load (1–9)	7.47 (0.94)	7.33 (1.03)	7.57 (1.01)	7.87 (0.86)
Learning efficiency	−0.37 (0.72)	0.02 (0.99)	0.04 (0.97)	0.30 (0.86)
Learning satisfaction (17–85)	62.20 (10.36)	61.37 (11.15)	62.60 (10.52)	66.67 (9.85)
Percentage of fixation duration on slides area	89.91 (13.48)	89.03 (8.17)	83.26 (12.93)	78.93 (18.25)
Percentage of fixation duration on instructor area	9.69 (12.96)	9.85 (7.23)	15.87 (12.73)	19.99 (17.87)

3.1. Learning performance

To test the difference in learning performance across the four conditions (H1), two repeated-measures ANOVAs were conducted on retention and transfer scores respectively. Concerning retention tests, significant difference was found across the four conditions, F(3, 87) = 9.39, p < .001, η_p² = 0.25. Further Bonferroni post hoc tests (see Fig. 5) showed that students in the Imitating gestures condition showed better retention performance than in the No gestures and Observing gestures conditions. Additionally, students in Imagining gestures condition gained slightly higher retention scores than in the No gestures condition. Concerning transfer scores, the results also showed a significant difference, F(3, 87) = 10.58, p < .001, η_p² = 0.27. Specifically, students in Imitating gestures condition showed the highest transfer scores among the four conditions (see Fig. 5).

3.2. Cognitive load

Inconsistent with H2, the repeated-measures ANOVA result showed no significant difference across the four conditions on cognitive load, F(3, 87) = 2.18, p = .097, η_p² = 0.07. It indicated that learning strategies did not influence students’ cognitive load.

3.3. Learning efficiency

To test the difference in learning efficiency across the four conditions (H3), one repeated-measures ANOVA was conducted. The result showed a significant difference, F(3, 87) = 3.42, p = .021, η_p² = 0.11. Further Bonferroni post hoc tests revealed that students in the Imitating gestures condition showed higher learning efficiency than No gestures condition (MD = 0.65, p = .037), which was slightly consistent with H4. What’s more, compared to No gestures condition, the other three all improved students’ learning efficiency indicated by a value greater than zero.

3.4. Learning satisfaction

To test the difference in learning satisfaction across the four conditions (H4), one repeated-measures ANOVA was conducted. The results indicated that there was a slightly significant difference across the four conditions, F(2.17, 62.80) = 2.92, p = .057, η_p² = 0.09 (Greenhouse-Geisser corrected for non-sphericity). Further Bonferroni post hoc tests revealed that students in the Imitating gestures condition reported higher satisfaction than in the Observing gestures condition (MD = 5.30, p = .020), which was slightly consistent with H5.

3.5. Attention allocation

To analyze the differences in students’ attention allocation (H5), Friedman tests were conducted on the percentage of fixation duration on two AoIs (slides and instructor areas) separately. Significant differences in the percentage of fixation duration on slides area and instructor area across the four condition were observed (respectively, χ² = 22.12, p < .001; χ² = 19.80, p < .001). Further pairwise comparison showed that students in the Imitating gestures and Imagining gestures conditions spent less percentage fixation duration on slides and more on instructor than in the other two conditions (see Fig. 6 and Fig. 7), which partly support H6.

3.6. The moderation effect of spatial ability

To verify H6, which was the moderation effect of spatial ability, a series of repeated-measures ANOVAs were conducted. Significant interaction effect was found for transfer performance, F(3, 84) = 4.53, p = .005, η_p² = 0.14; but not for retention performance, cognitive load, learning efficiency, learning satisfaction, and attention allocation (ps > .05). Further Bonferroni post hoc tests were used to interpret the significant interaction. Figure 8 showed that low-spatial-ability students under the Imitating gestures condition showed the best transfer performance among the four conditions. Additionally, the Imagining gestures condition also performed better than No gestures condition. However, for high-spatial-ability students, no difference in transfer performance across the four conditions was observed. The above results partly supported H7.

This study aimed to compare the effects of observing, imagining, and imitating the instructor’s depictive gestures on learning from instructional videos, assessed by learning performance, cognitive load, learning efficiency, and learning satisfaction, and attention allocation. The results confirmed the significant benefits of imitating the instructor’s depictive gestures (e.g., improving learning performance, learning efficiency, and satisfaction). Moreover, the moderation role of spatial ability on transfer performance was observed.

As expected, this study found a prominent role in imitating the instructor’s depictive gestures in learning from instructional videos. Specifically, compared with the passive learning strategies (i.e., No gestures and Observing gestures conditions) and imagination strategies, students showed better learning performance (i.e., retention and transfer), learning efficiency, and reported higher learning satisfaction when they used the imitation strategy to learn a video with the instructor’s depictive gestures. These results were consistent with previous studies (Tellier, 2008; Baills et al., 2018; García-Gámez et al., 2021), which can be explained by the two processes involved in imitation: observation and execution of gestures. Firstly, imitation of gestures activated students’ mirror neuron system when both observing and when executing, which not only provided semantic information but also led to a joint understanding between the instructor and students, thus resulting in the better learning performance and learning satisfaction (Mainieri et al., 2013; Holle et al., 2008; Brucker et al., 2015). Secondly, the results were supported by embodied cognition theory, that was, the execution of gestures influenced the cognitive process (Wilson, 2002). Furthermore, imitating gestures can help students externalize mental representations to expand their limited cognitive resources, which was beneficial for learning efficiency (Yang et al., 2021b). In other words, students can achieve better learning performance with the same amount of mental effort.

It is remarkable that we did not find an expected advantage of imagination strategy in learning performance, learning efficiency, and learning satisfaction. There are several possible explanations for the results. Firstly, although imagination strategy can promote the selection, organization, and integration of learning materials, this process may exceed the limited cognitive capacity, resulting in a higher cognitive load. As a result, its positive effect on learning may be canceled out. Another interpretation may be related to students’ lower prior knowledge experience. Previous studies revealed that the effect of imagination was likely to be moderated by students’ prior knowledge experience (Cooper et al., 2001; Yang et al., 2021b). To be specific, students with higher knowledge experience may find the use of imagination strategy beneficial to their learning because their schema freed up working memory sufficiently to enable them to perform imaginative-related mental operations. In contrast, students with low prior knowledge experience may find that attempting to imagine hinders their learning because they lack the prerequisite schema required to imagine elements and relationships. In this study, the mean score of the prior knowledge test was only about 20% of the total score, indicating that students’ prior knowledge experience was relatively low, so the benefits of imagination strategy have not been verified.

With regard to attention allocation, the eye movement results showed that when using imagination and imitation strategies to learn videos with the instructor’s depictive gesture, learners allocated more attention to the instructor and less to the learning materials than using other strategies. However, the attention distraction did not reduce learning outcomes. Instead, the application of imitation strategy led to the best learning outcomes (i.e., learning performance, learning efficiency, and learning satisfaction). A possible interpretation of this result has to do with this kind of split attention being meaningful. Based on the dual channels principle of multimedia learning, the instructor used depictive gestures to present semantic information in students’ visual channels and completed teaching activities together with their verbal expressions (Paas & van Merriënboer, 1994; Mayer & Fiorella, 2022; Congdon et al., 2017). From this perspective, the instructor’s depictive gestures can avoid cognitive overload, thus releasing cognitive resources for better acquisition of new knowledge. Additionally, although the instructor’s depictive gestures attracted more attention from students, the information expressed by such gestures was consistent with the learning materials, In this case, students used active learning strategies (i.e., imagination and imitation) to process the instructor’s depictive gestures, which was beneficial for students to comprehend the learning content deeply. Therefore, paying more attention to the instructor did not hinder learning.

More importantly, this study confirmed that spatial ability moderated the effect of learning strategy on transfer performance. Specifically, the results revealed that low-spatial-ability students showed better transfer performance when viewing videos with active learning strategies (i.e., imagining and imitating the instructor’s gestures) than passively viewing, which was in line with the findings of Brucker et al. (2015). Their experimental results revealed that low-spatial-ability students performed better in the well-designed condition than in the poor-designed condition when they were required to understand and classify the fish movement patterns; while for high-spatial-ability students, the design of materials had no effect. According to the ability-as-compensator hypothesis, low-spatial-ability students were insufficiently equipped for coping with learning materials and the instructor’s depictive gestures, so active learning strategies were necessary to reduce the conflict and facilitate learning performance, especially for transfer performance. In contrast, high-spatial-ability students may not require optimal instruction (Höffler, 2010; Brucker et al., 2015). However, contrary to the results of Brucker et al. (2022), we found the benefits of imitating gestures for low-spatial-ability students. A possible explanation was that the gesture instruction in the study of Brucker et al. was not clear, and students were only required to make gestures in a way that they thought to help understand the fish movement patterns, which was confusing for low-spatial-ability students, so the positive effects of making gesture were not observed. In this study, we explicitly asked students to make the same gestures as the instructor, so it turned out to be more effective for them. In addition, we failed to find the moderation effects on retention performance, cognitive load, learning efficiency, learning satisfaction, and attention allocation, which may be due to the small sample of the low-spatial-ability (n = 16) and high-spatial-ability (n = 14) groups.

There are some limitations to be further considered. First of all, we assessed cognitive load using a single-item mental effort measure to keep the burden on participants low (e.g., Hefter & Berthold, 2020). However, the scale was self-reported after learning, which indicated we cannot specify which type of cognitive load affects learning. By contrast, electroencephalography provides real-time measures of neural activity associated with cognitive load (Pi et al., 2021). Thus, a mixed way can be considered to examine the cognitive load in the future. Secondly, in order to eliminate the interference of students’ personality characteristics on the results, the study adopted a within-subjects design, in which each student had to experience four experimental conditions. Although we provided them with 5-min rest time after each condition, this inevitably led to a fatigue effect, as indicated by the higher cognitive load scores reported by students. Therefore, future work is needed to reduce the sense of fatigue by increasing rest time. Finally, the sample size in this study was small, especially for low-spatial-ability (n = 16) and high-spatial-ability (n = 14) students. Therefore, the generalisability of the findings is limited. Future studies can consider repeating the findings of this research in a larger sample.

In conclusion, this study has been one attempt to thoroughly compare the effects of learning strategies and found the significant benefits of imitating the instructor’s depictive gestures and slight positive effects of imagining gestures. When using imagination and imitation strategies to learn videos with the instructor’s depictive gestures, students’ attention was distracted by the instructor. Furthermore, the active learning strategies may be more effective for low-spatial-ability students than for high-spatial-ability students. The results lead to a strong recommendation for educational practice when using instructional videos: (1) when watching videos with the instructor’s depictive gestures, students are encouraged to use imitation strategy in preference to imagination strategy, and passive observation is not recommended; and (2) low-spatial-ability students are encouraged to use active learning strategies (i.e., imagining and imitating the instructor’s depictive gestures) to improve transfer performance.

Competing Interests:

No potential conflict of interest was reported by the authors.

Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: A view from students’ and teachers’ gestures. Journal of the Learning Sciences, 21(2), 247–286.
Alibali, M. W., Nathan, M. J., Wolfgram, M. S., Church, R. B., Jacobs, S. A., Johnson Martinez, C., & Knuth, E. J. (2013). How teachers link ideas in mathematics instruction using speech and gesture: A corpus analysis. Cognition and Instruction, 32(1), 65–100.
Baills, F., Suárez-González, N., González-Fuente, S., & Prieto, P. (2018). Observing and producing pitch gestures facilitates the learning of mandarin Chinese tones and words. Studies in Second Language Acquisition, 41(1), 33–58.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall.
Brucker, B., Ehlis, A. C., Häußinger, F. B., Fallgatter, A. J., & Gerjets, P. (2015). Watching corresponding gestures facilitates learning with animations by activating human mirror-neurons: An fNIRS study. Learning and Instruction, 36, 27–37.
Brucker, B., de Koning, B., Rosenbaum, D., Ehlis, A. C., & Gerjets, P. (2022). The influence of gestures and visuospatial ability during learning about movements with dynamic visualizations - An fNIRS study. Computers in Human Behavior, 129, article 107151.
Castro-Alonso, J. C., Ayres, P., & Paas, F. (2014). Learning from observing hands in static and animated versions of non-manipulative tasks. Learning and Instruction, 34, 11–21.
Congdon, E. L., Novack, M. A., Brooks, N., Hemani-Lopez, N., O’Keefe, L., & Goldin-Meadow, S. (2017). Better together: Simultaneous presentation of speech and gesture in math instruction supports generalization and retention. Learning and Instruction, 50, 65–74.
Cooper, G., Tindall-Ford, S., Chandler, P., & Sweller, J. (2001). Learning by imagining. Journal of Experimental Psychology: Applied, 7(1), 68–82.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191.
Fiorella, L., & Mayer, R. E. (2015). Eight ways to promote generative learning. Educational Psychology Review, 28(4), 717–741.
Fiorella, L., & Mayer, R. E. (2016). Effects of observing the instructor draw diagrams on learning from multimedia messages. Journal of Educational Psychology, 108, 528–546.
García-Gámez, A. B., Cervilla, Ó., Casado, A., & Macizo, P. (2021). Seeing or acting? The effect of performing gestures on foreign language vocabulary learning. Language Teaching Research. Advance online publication.
Goldin-Meadow, S. (2014). How gesture works to change our minds. Trends in Neuroscience and Education, 3(1), 4–6.
Hefter, M. H., & Berthold, K. (2020). Preparing learners to self-explain video examples: text or video introduction?. Computers in Human Behavior, 110, article 106404.
Höffler, T. N. (2010). Spatial ability: Its influence on learning with visualizations - a meta-analytic review. Educational Psychology Review, 22(3), 245–269.
Holle, H., Gunter, T. C., Rüschemeyer, S. A., Hennenlotter, A., & Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. NeuroImage, 39(4), 2010–2024.
Ianì, F., Burin, D., Salatino, A., Pia, L., Ricci, R., & Bucciarelli, M. (2018). The beneficial effect of a speaker’s gestures on the listener’s memory for action phrases: The pivotal role of the listener’s premotor cortex. Brain and Language, 180–182, 8–13.
James, K.H., & Swain, S.N. (2011). Only self-generated actions create sensori-motor systems in the developing brain. Developmental Science, 14, 673–678.
Leahy, W., & Sweller, J. (2004). Cognitive load and the imagination effect. Applied Cognitive Psychology, 18(7), 857–875.
Lee, H. (2007). Instructional design of web-based simulations for learners with different levels of spatial ability. Instructional Science, 35, 467–479.
Leopold, C., Mayer, R. E., & Dutke, S. (2019). The power of imagination and perspective in learning from science text. Journal of Educational Psychology, 111(5), 793–808.
Mainieri, A. G., Heim, S., Straube, B., Binkofski, F., & Kircher, T. (2013). Differential role of the Mentalizing and the Mirror Neuron system in the imitation of communicative gestures. NeuroImage, 81, 294–305.
Marcus, N., Cleary, B., Wong, A., & Ayres, P. (2013). Should hand actions be observed when learning hand motor skills from instructional animations? Computers in Human Behavior, 29, 2172–2178.
Mayer, R. E. & Fiorella, L. (2022). The Cambridge handbook of multimedia learning (3^rd ed.). Cambridge University Press.
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press.
Moning, J., & Roelle, J. (2021). Self-regulated learning by writing learning protocols: Do goal structures matter? Learning and Instruction, 75, article 101486.
Paas, F. G., & van Merriënboer, J. J. (1994). Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review, 6, 351–371.
Pi, Z., Zhang, Y., Yang, J., Hu, W., & Yang, H. H. (2019). All roads lead to Rome: Instructors’ pointing and depictive gestures in video lectures promote learning through different patterns of attention allocation. Journal of Nonverbal Behavior, 43(4), 549–559.
Pi, Z., Zhang, Y., Yu, Q., Zhang, Y., Yang, J., & Zhao, Q. (2021). Neural oscillations and learning performance vary with an instructor’s gestures and visual materials in video lectures. British Journal of Educational Technology. Advance online publication.
Ping, R., Goldin-Meadow, S., & Beilock, S. L. (2014). Understanding gesture: Is the listener’s motor system involved? Journal of Experimental Psychology: General, 143(1), 195–204.
Rumiati, R. I., & Bekkering, H. (2003). To imitate or not to imitate? How the brain can do it, that is the question! Brain and Cognition, 53(3), 479–482.
Son, J. Y., Ramos, P., DeWolf, M., Loftus, W., & Stigler, J. W. (2018). Exploring the practicing-connections hypothesis: Using gesture to support coordination of ideas in understanding a complex statistical concept. Cognitive Research: Principles and Implications, 3, 1–13.
Tellier, M. (2008). The effect of gestures on second language memorization by young children. Gesture, 8, 219–235.
van Gog, T., & Paas, F. (2008). Instructional efficiency: Revisiting the original construct in educational research. Educational Psychologist, 43, 16–26.
Willems, R. M., & Hagoort, P. (2007). Neural evidence for the interplay between language, gesture, and action: A review. Brain and Language, 101(3), 278–289.
Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–636.
Wong, M., Castro-Alonso, J. C., Ayres, P., & Paas, F. (2018). Investigating gender and spatial measurements in instructional animation research. Computers in Human Behavior, 89, 446–456.
Yang, J., Liu, C., Zhang, Y., Yu, Q., & Pi, Z. (2021a). The teacher’s eye gaze in university classrooms: evidence from a field study. Innovations in Education and Teaching International. Advance online publication.
Yang, J., Zhang, Y., Xu, K., & Pi, Z. (2021b). The effects of learning strategies on video learning: imagination, drawing and self-explanation strategies. E-Education Research, 42(10), 40–47.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

The effects of observing, imagining, and imitating the instructor’s depictive gestures on learning from instructional videos

Status:

Version 1

Abstract

Figures

1. Introduction

1.1. The effects of depictive gestures on learning from instructional videos

1.2. Comparisons of observation, imagination, and imitation strategies

1.3. The moderation role of spatial ability

1.4. The present study

2. Method

2.1. Participants and design

2.2. Instructional videos

2.3. Measurements

2.3.1. Spatial ability test

2.3.2. Prior knowledge test

2.3.3. Learning performance tests

2.3.4. Cognitive load scale

2.3.5. Learning efficiency

2.3.6. Learning satisfaction scale

2.4. Eye movement data collection and analysis

2.5. Procedure

3. Results

3.1. Learning performance

3.2. Cognitive load

3.3. Learning efficiency

3.4. Learning satisfaction

3.5. Attention allocation

3.6. The moderation effect of spatial ability

4. Discussion

Declarations

References

Additional Declarations

Status:

Version 1