This study took place at a technological university. Its purpose was to explore whether the understanding of metacognitive strategies could improve college-level EFL learners’ listening comprehension through the application of an approach using instruction assessed by YouTube and podcast multimedia materials. Pretest and posttest scores were compared, and a Metacognitive Awareness Listening Questionnaire was conducted at the end of the semester.
Participants
The participants in the study were enrolled in a TOEIC remedial courses due to failure in their freshman English course. These students needed to pass this course to fulfill their graduation threshold. The course curriculum is intended to bring students to the level where they could score 550 points in the TOEIC. The evaluation criteria included online e-platform assignments and mock tests as well as a midterm and a final exam. Participating students were randomly assigned to the control (33 students, standard instruction) or experimental (50 students, multimedia) group.
Design of the study
The course objectives were focused on the development of students’ English proficiency and test-taking skills to reach the goal of scoring 550 points on the TOEIC as the ultimate result. The course curriculum includes the study of a test preparation text as its main materials; in this text, items from different parts of the TOEIC are analyzed and demonstrated. Furthermore, the curriculum includes self-evaluated online assignments to constantly engage students with mock test questions to familiarize the participants with the TOEIC questions.
The instructors for this course have the freedom to design their own quizzes, reports, or assignments. The course for the experimental group was planned with the use of multimedia, which could potentially enhance students’ listening comprehension. In addition, the students were informed about the concept of metacognitive strategies and learning strategies for coping with listening comprehension tests. The students in the control group were provided instruction through the regular curriculum design, with the same standard textbook and the same standard online assignments and tests. Both groups had midterms and final examinations with the same test questions from the Easy Test online testing platform purchased by the language center; this platform also provides online mock tests. Easy Test provides users with a variety of learning content, including TOEIC courses, video learning, vocabulary, grammar, writing, and so on. Its materials are suitable for English learners at all levels. The interface to Easy Test’s system is simple and does not require additional software. In addition, mock tests for multiple language proficiency exams are provided, including TOEIC, TOEFL, and IELTS. The learners can train on mock tests to develop their problem-solving skills and test-taking speed, and their training could increase learners’ familiarity with the test questions. The online assignments embedded in the curriculum were provided by the Vocabulary & Grammar Drills e-platform, which contains many vocabulary and grammar exercises related to TOEIC type of questions.
This study focuses on the improvement of the participants’ listening comprehension through multimedia-infused instruction with the assistance of metacognitive strategies. The five major components of Anderson’s model of metacognitive strategy development were presented in instruction for the experimental group, namely, preparing and planning for learning; selecting and using learning strategies, monitoring strategy use, orchestrating various strategies, and evaluating strategy use and learning were reviewed (Tham, 2018). After the introduction of each component, the participants took a listening comprehension test conducted through the use of YouTube and podcasts. These test materials were carefully selected in relation to the themes of the TOEIC, and 10 multiple questions or true/false questions were designed according to each YouTube video content and Podcast audio files. There were a total of eight multimedia listening comprehension tests throughout the semester. This study extended over a total of were a total of 18 class meetings. During the 10 intervention classes, the instructor spent about 20 minutes explaining the concepts of the listening strategies and the components of metacognitive strategies. Following this, a comprehension question test was conducted. The results of the listening comprehension section from both groups’ midterms and final exams were compared and analyzed. At the end of the study, the participants in the experimental group were administered the Metacognitive Awareness Listening Questionnaire (MALQ), developed by Vandergrift, Goh, Mareschal, and Tafaghodtari (2006) to explore their opinions on the effectiveness of the strategies that had been implemented.
Instrument
The MAL was administered to evaluate language learners’ awareness of and engagement in second language listening comprehension. The MALQ was also designed to serve as a self-assessment instrument for estimating learners’ awareness of the listening process and to evaluate how well learners understand their use of strategies during the listening comprehension tasks (Vandergrift, Goh, Mareschal and Tafaghodtari, 2006). The MALQ was developed in relation to Flavell’s (1979) model of metacognitive knowledge.
Metacognitive knowledge is a fundamental element in many cognitive strategies associated with the use of language, such as delivering messages through oral communication, engaging in oral persuasion, oral comprehension, reading comprehension, and writing. This is essential to language acquisition and to language learners’ self-instruction (Wenden, 1999).
Data Collection and Analysis
The control and the experimental groups went through the same curriculum design with the same textbook. The two groups had a very similar academic background. The only difference between the groups was that the experimental group was provided with the metacognitive strategies. After each brief introduction of the metacognitive knowledge for language learning, the students in the experimental group were given a test with listening materials, drawn four times from YouTube and four times from podcasts. The purpose of using multimedia listening exercises was to determine out whether the students were making progress with the assistance of using metacognitive strategies and whether the differences between these two types of multimedia had an impact on the students’ listening comprehension.
At the end of the semester, the participants in the experimental group were administered the MALQ to understand the students’ learning patterns. The control group’s test results in the listening session from their midterm and final exams were compared with the results from the experimental group. In the data collection, the participants in the experimental group were assessed for listening comprehension of the multimedia materials. The two sources of multimedia used differed in that the YouTube materials focused on visual effects, including displaying messages with audio and subtitle functions. Podcasts, however, provide audio exclusively. Metacognitive strategies were presented before the comprehension tests.
Table 1. Descriptive statistics analysis for the YouTube assessment
|
N
|
Min.
|
Max.
|
Mean
|
SD
|
1st test
2nd test
3rd test
4th test
|
51
51
51
51
|
20
30
30
30
|
90
90
90
90
|
61.37
66.08
66.27
70.39
|
17.552
15.631
14.137
14.417
|
The mean scores of the four YouTube listening comprehension tests were 61.37, 66.08, 66.27, and 70.39, showing an upward trend, while the standard deviations gradually shrank. This indicates that the students’ scores were converging on a mean and becoming more stable.
Table 2. Paired sample t -test for YouTube assessment
|
|
mean
|
SD
|
Std. error mean
|
Confidence interval
|
t
|
df
|
Sig.
|
Lower limit
|
Upper limit
|
Pair 1
|
1st & 2nd test
|
−4.076
|
21.941
|
3.072
|
−10.877
|
1.465
|
−1.532
|
50
|
.132
|
Pair 2
|
1st & 3rd test
|
−4.902
|
23.270
|
3.258
|
−11.447
|
1.643
|
−1.504
|
50
|
.139
|
Pair 3
|
1st & 4th test
|
−9.020
|
24.270
|
3.398
|
−15.846
|
−2.194
|
−2.654
|
50
|
.011
|
Pair 4
|
2nd & 3rd test
|
−.196
|
20.148
|
2.821
|
−5.863
|
5.471
|
−.069
|
50
|
.945
|
Pair 5
|
2nd & 4th test
|
−4.314
|
20.905
|
2.927
|
−10.193
|
1.566
|
−1.474
|
50
|
.147
|
Pair 6
|
3rd & 4th test
|
−4.118
|
15.515
|
2.172
|
−8.481
|
.246
|
−1.895
|
50
|
.064
|
A paired sample t-test was conducted to analyze data from the first test to the fourth test to identify any significant differences between any two tests. Only significant differences with p <.05 between the two tests are shown in the paired comparisons. As seen in Table 2, only pair 3 (the first test and the fourth test) indicated a significant difference (.011 < 0.05). The mean score between the first test and the fourth test progressed from 61.34 to 70.39, for an improvement rate of approximately 14.7 %.
Table 3. The descriptive statistics analysis for podcast assessments
|
N
|
Min.
|
Max.
|
Mean
|
SD
|
1st test
|
51
|
15
|
100
|
56.96
|
19.977
|
2nd test
|
51
|
20
|
100
|
65.78
|
18.121
|
3rd test
|
51
|
50
|
95
|
75.10
|
12.668
|
4th test
|
51
|
60
|
100
|
85.29
|
10.169
|
The data in Table 3, for the podcast assessments, indicate that the mean scores gradually progressed in each test. Moreover, the standard deviations became gradually smaller, indicating that the students’ performances were moving toward stability.
Table 4. Paired sample t-test for podcast assessment
A paired sample t-test was conducted to analysis the data from the first test to the fourth test to identify whether any significant difference appeared between any of the two tests. Only significant differences of p <.05 between the two tests are shown among the six pairs compared. From the analysis of the paired sample t-test for these six pairs in Table 4, each pair showed a significantly difference (.000 <.05). The analysis of the descriptive statistics, it was evident that the test scores were increasing every time. The mean score grew from the first test at 56.96 to the fourth test at 85.29. The improvement rate was approximately 49.7%.
Analysis of the Metacognitive Awareness Listening Questionnaire
The MALQ measures five factors, namely, problem-solving, planning and evaluation, mental translation, individual prior knowledge, and directed attention. The reliability of the Likert scales has been identified to have Cronbach’s alpha coefficients of.74 for problem-solving, 0.75 for planning and evaluation, 0.78 for translation, 0.74 for individual prior knowledge, and 0.68 for directed attention, respectively (Rahimi & Katal, 2012).
The participants in the experimental group completed a MALQ at the end of the study to identify the most and least frequently used strategies in the multimedia listening comprehension test and the TOEIC mocking listening test. Descriptive statistics for the questionnaire indicate that the most frequently used strategy was “I use the words I understand to guess the meaning of the words I do not understand” (question 5). The mean score for this item on the questionnaire was 4 out of 5, and the second most frequently used strategies were “I use my experience and knowledge to help me understand” (question 9) and “I focus harder on the text when I have trouble understanding” (question 2). The mean scores for these two questions were respectively 3.923 and 3.897.
The analysis also revealed that the least frequently used strategy was “I do not feel nervous when I listen to English” (question 15). The mean score for the question was 2.923. Therefore, the score indicated that the students could be anxious and tensed when taking the comprehension tests. The second-least-used skills were “After listening, I think back to how I listened, and about what I might do differently next time” (question 14) and “I find that listening in English is more difficult than reading, speaking, or writing in English” (question 3). The mean scores for these two items were 2.974 and 3.128, respectively.
In the questionnaire, items 1 to 6 describe problem-solving strategies. Items 7 to 11 relate to planning and evaluation strategies. Items 12 to 14 relate to mental translation strategies, and items 15 to 17 cover individual prior knowledge strategy. Finally, items 18 to 21 describe the directed attention strategy. The results of the analysis show that the students were more influenced by the problem-solving and planning and evaluation strategies. Furthermore, they were less influenced by the mental translation and the individual prior knowledge strategies when coping with the listening comprehension test.
Analysis of the midterm exam and the final exam in the control group
The purpose of this t-test was to analyze data in the listening comprehension tests from the midterm exam to the final exam in the control group to identify any significant differences.
Table 5. Paired sample statistical analysis for listening comprehension tests in the control group
|
Mean scores
|
N
|
SD
|
Std. error mean
|
Midterm listening test
|
160.50
|
33
|
42.279
|
7.360
|
Final listening test
|
211.00
|
33
|
57.682
|
10.041
|
Table 6. Correlated sample test in the control group
|
N
|
Correlation
|
Sig.
|
Listening test scores from midterm to final exam
|
33
|
.685
|
.000
|
Table 7. Paired sample t-test in the control group
|
Mean
|
SD
|
Std. error mean
|
Confidence interval
|
t
|
df
|
Sig.
|
Lower
limit
|
Upper limit
|
Listening test scores from midterm to final exam
|
−50.500
|
42.122
|
7.333
|
−65.436
|
−35.564
|
−6.887
|
32
|
.000
|
The mean score for the listening test in the midterm exam from the control group was 160.50, with a standard deviation of 42.279. In addition, the mean score for the listening test in the final exam from the control group was 211.00, with a standard deviation of 57.682. The mean scores progressed from 160.50 to 211.00, with an improvement rate of 31.46%.
Table 6 presents a correlation of 0.685 and p =.000 <.05, which indicates a significant correlation. Furthermore, the paired sample t-test compared the listening scores between the midterm exam and the final exam in the control group, as shown in Table 7), which indicated a t value −6.887 and a two-tailed p value of.000 <.05, which indicated significance. Thus, the results showed a significant difference.
Analysis of the midterm exam and the final exam in the experimental group
Table 8. Paired sample statistical analysis for listening comprehension tests in the experimental group
|
Mean scores
|
N
|
SD
|
Std. error mean
|
Midterm listening test
Final listening test
|
185.57
244.53
|
50
50
|
36.174
61.749
|
5.116
8.733
|
Table 9. Correlated sample test in the experimental group
|
N
|
Correlation
|
Sig.
|
Listening test scores from midterm to final exam
|
50
|
.565
|
.000
|
Table 10. Paired sample t-test in the experimental group
|
Mean
|
SD
|
Std. error mean
|
Confidence interval
|
t
|
df
|
Sig.
|
Lower
limit
|
Upper limit
|
Listening test scores from midterm to final exam
|
−58.960
|
50.945
|
7.205
|
−73.438
|
−44.482
|
−8.184
|
49
|
.000
|
From the descriptive statistical analysis, the mean score of the listening test in the midterm exam from the experimental group was 185.57, with a standard deviation of 36.174. In addition, the mean score of the listening test in the final exam in the experimental group was 244.53, with a standard deviation of 61.749. The mean scores progressed from 185.57 to 244.53, with an improvement rate of 31.77 %.
Table 9 showed that a correlation coefficient of 0.565 and p =.000 <.05 which indicated a significant correlation. Furthermore, a paired sample t-test was conducted to compare the listening scores between the midterm exam and the final exam in the experimental group in table 10, which revealed the t value −8.184 and two-tailed significant p =.000 <.05. Therefore, the result showed a significant value.