Result and discussion of research question one
The students were given pretest and posttest to measure their learning outcomes before and after the course. Accordingly, this section presents the students’ test results gained from the pretest and posttest, which were analyzed using the Wilcoxon signed-rank sum test.
Table 2 indicates that the students’ basic Japanese proficiency test mean value improved from 6.13 (out of a full score of 12, SD = 2.09) to 10.00 (SD = 1.73). Additionally, a significant difference (P = 0.001) was noted in the distribution of basic Japanese scores. From this result, we can deduce that the students’ basic Japanese knowledge improved during this CLIL course.
Table 2 Wilcoxon signed-rank test results of pre-post Basic Japanese Proficiency Test Score (N = 16)
Test
|
Mean
|
Median
|
SD
|
z
|
P value
|
Pre-basic
|
6.13
|
6.00
|
2.09
|
3.305
|
0.001
|
Post-basic
|
10.00
|
10.00
|
1.73
|
Note. * p < 0.05; ** p < 0.01.
Table 3 indicates that the students’ content knowledge test mean value improved from 2.00 (out of a full score of 6, SD = 1.51) to 4.62 (SD = 1.50). Additionally, a significant difference (P = 0.001) was noted in the distribution of content knowledge scores. From this result, we can understand that the students’ content knowledge improved during this CLIL course.
Table 3 Wilcoxon signed-rank test results of pre-post Content Knowledge Test Score (N = 16)
Test
|
Mean
|
Median
|
SD
|
z
|
P value
|
Pre-content
|
2.00
|
2.00
|
1.51
|
3.424
|
0.001
|
Post-content
|
4.62
|
5.00
|
1.50
|
Note. * p < 0.05; ** p < 0.01.
Table 4 indicates that students’ writing test mean value improved from 4.19 (out of a full score of 12, SD = 1.00) to 9.38 (SD = 1.64). Additionally, a significant difference (P = 0.000) was noted in the distribution of content knowledge scores. From this result, we can understand that the students’ writing skills improved during this CLIL course.
Table 4 Wilcoxon signed-rank test results of pre-post of Writing Test Score (N = 16)
Test
|
Mean
|
Median
|
SD
|
z
|
P value
|
Pre-writing
|
4.19
|
4.50
|
1.00
|
3.535
|
0.000
|
Post-writing
|
9.38
|
9.50
|
1.64
|
Note. * p < 0.05; ** p < 0.01.
In sum, it can be understood that the CLIL course designed by FPI has positive effects on language knowledge, writing skills, and content knowledge. In line with this research finding, previous studies also show that course designed by FPI has positive impacts on students’ learning outcomes. Lo et al. (2018) found out that by using FPI in a flipped course, students develop scholastic ability, Chinese language ability, and learn how to generate and organize ideas.
Because CLIL is characterized by using a foreign language to teach specialized knowledge, the goal is to improve both the foreign language and the specialized knowledge, which Coyle (2010) notes is not an easy challenge to accomplish. Ennis (2015) also notes in his paper that CLIL courses require explaining what students do not know in a language they do not understand, which is a very challenging task. In addition, Agustín-Llach (2016) points out that there were no significant differences in student achievement between the CLIL and no-CLIL courses. So it can be seen that if the language and content are added together, there is a possibility that the desired learning goals will not be achieved. Although, this CLIL course designed using FPI guarantees both content and language learning outcomes.
However, regarding the SD in Table 4, the post-SD (1.64) was also higher than the pre-SD (1.00). Students’ writing skills were strengthened through this lesson, but the disparity between students also widened. Students’ writing skills were strengthened through this course, but the disparity between students also widened. Although the design for writing is effective for the whole course, the degree of understanding of writing content may differ due to individual differences. Allison, Cooley, Lewkowicz, and Nunan (1998) point out that higher education students undertake limited writing exercises in the classroom and require one-to-one writing instruction. In this course, although the teacher provided a worksheet to help students organize their writing ideas before they wrote, the teacher did not show students specific examples of writing. This may have resulted in students not understanding the use of the worksheet, thus causing an increase in personal differences due to their own comprehension. This is also expressed in the dialogue.
Example 1. Use the worksheet to make the respective Johari window.
Group 1
Scene 1
G1-S2: Have you all finished writing? I have finished writing all of them.
G1-S3: Do I need to continue writing? Or is it enough to write only one part of it?
G1-S1: You need to write all of them.
G1-S2: I think I should have drawn this part a little bigger.
(4 seconds)
G1-S3: Well, then my first part should be bigger too.
Scene 2
G1-S3: Eh? Which position should this be written in?
G1-S2: On love, something
G1-S3: I do not know.
G1-S1: (laughs) love men and women or both.
G1-S3: Eh? This is not it? It is the second one, right?
G1-S2: Position 2, the second one is empty.
Among the four students in Group 1, one student was not clear about the use of the worksheet. Moreover, this student asked how to use the worksheet two times during the discussion. The student who asked the question finally completed the worksheet after constantly checking with the other group members. This is consistent with the results in Table 4. Students experienced some differences in understanding when using the worksheet.
Group 2
G2-S2: I want to write myself やさしい(kind)
G2-S4: That I am かわいい(lovely)
G2-S2: This should be written in the part that you know that others do not know, right? Is it written in position 3?
G2-S1: Yes, here, you know, but others do not know.
One student in Group 2 also had a question about using the worksheet. The group members gave clear answers. Again, this group's discussion is consistent with the results in Table 4 in that there are individual differences among students.
Group 3
G3-S4: Been on a diet.
(laughing)
G3-S3: But you are not really doing the exercise to lose weight either.
G3-S1: Yeah, I did not.
G3-S3: Been studying English.
G3-S1: Already written.
(3seconds)
G3-S3: Well, we are at an impasse.
(…)
G3-S2: So what should be written in the third position?
G3-S4: It does not have to be a bad aspect. It can be a good one too, right?
G3-S2: But the good aspects are already known to others.
(8 seconds)
G3-S3: OK. Once again, there was an impasse.
It can be seen from the conversation that one student in Group 3 also raised questions about using the worksheet. However, the problem raised was not solved in the end. The discussion in Group 3 also showed consistency with the results in Table 4. In particular, asking questions that are not answered can exacerbate personal differences among students.
Group 4
G4-S1: The second position is to write what others know and what I do not know, right?
(5 seconds)
G4-S3: Yeah, the second position is what others know and what you do not know yourself.
G4-S1: This is the Johari window in … others know that I do not know. OK, then a little more specific is…
G4-S4: It is your interest and what you like to do, and so on.
G4-S1: Like habits! It is one of those little habits that you know but others do not. And then there is the kind of mantra you have, you do not realize it, but others do.
G4-S4: Yeah, verbal words must be known to others because you always say them.
G4-S2: Oh, I see, some little gestures and habits!
G4-S1: Yeah, Bad habits are something that others know about you, but you do not know about yourself.
G4-S4: Okay, so write about some bad habits in this position.
A student in Group 4 also asked a question about the worksheet. The group members responded positively, and the student who asked the question eventually solved the problem. It is also evident from the conversations in Group 4 that there were personal differences among the students, consistent with the results in Table 4.
From the conversations, it can be seen that all four groups of learners had problems with not knowing how to use the worksheet or what kind of content should be recorded in which position in the worksheet. Furthermore, it can be seen that each group addressed the issues raised by the members in different ways. Groups 1,2,4 all solved this problem through discussion, but group 3 did not solve these problems in the end either. This is consistent with the results in Table 4. Students experienced some differences in understanding when using the worksheet, and problems may arise that cannot be resolved in the group, as demonstrated by the conversations in Group 3. This can also cause individual differences between students to grow. Students who already understand how to use worksheets can organize their writing ideas better through the use of the worksheet, while students who do not understand and cannot get answers from the group members become more confused. When designing a course to solve this problem, it is necessary to understand the learning situation of each student and provide support to the individual. In addition, should evaluate the ease of use of the worksheets before the formal course to improve the effectiveness of the worksheets.
Result and discussion of research question one of research question two
Table 5 indicates the students’ awareness of the designed FPI elements. Results on the questionnaire items for the awareness of FPI elements, all elements received high ratings of more than 4 points, showing that the course was designed to allow students to experience the FPI elements. The highest score is integration element (mean 4.53, SD 0.27), followed by activation element (mean 4.38, SD 0.32), followed by demonstration element (mean 4.33, SD 0.58) and application element (mean 4.33, SD 0.27), and the lowest score is problem-centered element (mean 4.09, SD 0.55).
Table 5 Descriptive Statistics of students’ awareness of the FPI elements (N = 16)
|
Min
|
Max
|
Mean
|
SD
|
Problem-Centered
|
3.25
|
5.00
|
4.09
|
0.55
|
Activation
|
3.80
|
4.80
|
4.38
|
0.32
|
Demonstration
|
3.00
|
5.00
|
4.33
|
0.58
|
Application
|
3.75
|
4.75
|
4.33
|
0.27
|
Integration
|
3.80
|
4.80
|
4.53
|
0.27
|
Notably, students did not show much variation in their awareness of the elements in table 5 However, as described in the course design section, the length of the course's application element was increased to provide students with more active learning opportunities. The application element accounted for 33% of the course time, and the integration element accounted for 22% of the course time. While the integration element received the highest score, the application element scored relatively low.
Borg (2006) points out that effective language teaching requires rich classroom diversity and an environment where students can increase their participation and the quality of their involvement. This course was designed to ensure that students participated in the activities, and the dialogue content during group discussions was analyzed to capture the quality of learning during participation time. The conversations related to the course content and Japanese language learning in the activity were coded. Of the 983 dialogues, 59 were related to lesson content, and 155 were related to Japanese language learning.
Example 2. Group discussion about contents and Japanese.
Group 1
G1-S1: I wrote “同じ”(same). I think it's all the same.
G1-S4: The teacher is asking you when.
G1-S3: You see, what the teacher's saying is originally you thought the Japanese were like this, and then it changed.
G1-S4: Yeah, that means when.
G1-S2: どんなときですか (When is it?)
G1-S1: No change. It's the same as I thought it would be.
G1-S3: No, the teacher asked when you had changed.
G1-S1: That is, the teacher was asking when it had changed, but I don't think it has.
G1-S4: That is when it has or has not changed.
G1-S1: It hasn't changed.
G1-S4: You just write about one thing that inspired you.
G1-S1: I am now writing about “同じ”(same).
(Writing and painting)
Group 1 had 443 conversations in the two learning activities, including 19 conversations that included knowledge content and 30 conversations about Japanese. Moreover, as demonstrated in this dialogue, most of the Japanese used were just words, and even when the sentences were in Japanese, they were repeating the topics of the questions presented. These conversations corroborate the results in Table 2. Students in Group 1 mostly use Japanese to confirm basic words, which can explain that the scores on the basic Japanese test have improved.
Group 2
Scene 1
G2-S2: What about this one?
G2-S3: Let's write another one
G2-S1: The third is to ask, at first you think so, and then after contact with them, what has changed, right?
G2-S2: Yeah. Just think of them for “真面目” (Seriously).
G2-S1: どんなときで(At what time). At what time was it? OK, then next.
G2-S2: Can this space be written down?
(5 seconds)
G2-S2: Just write keywords?
G2-S1: Yeah.
G2-S3: Right.
Scene 2
G2-S4: Eh, “見た”, is there a small “つ”?
G2-S3: No, there is no “つ.”
Group 2 had a total of 582 conversations during the two learning activities. Of these, 26 conversations were related to the knowledge content, and 77 conversations were related to Japanese. As shown in the example given for Group 2, as in Group 1, most of the conversations in Japanese consisted of sentences confirming the question topic, and most of the other conversations in Japanese were in words. However, there were many instances of confirmation and discussion of Japanese words in Group 2. These conversations corroborate the results in Tables 2 and 5. Although the students did not have the majority of conversations about knowledge content and Japanese in the group discussions during the application time, it was meaningful to confirm and consolidate the words learned during the group discussions.
Group 3
G3-S4: In fact, I would have liked to write about “曖昧”(ambiguous).
G3-S3: My theme is tenderness. 雲のような日本人(Cloudy Japanese). “雲” means cloud.
G3-S1: This is OK.
G3-S2: What about the specific things? Like they do something that makes you think they are like clouds.
G3-S3: That is, the characteristic is 優しい (gentle). Then they are polite. Specifically, the first impression was obtained from the Japanese drama. Then actual contact with the Japanese, the impression has not changed. なし(none).
G3-S2: OK
Group 3 had a total of 347 conversations in 2 learning activities. Eight of these were about knowledge content, and 34 were about Japanese. As with the conversations in Groups 1 and 2, most of the Japanese that appeared were words rather than complete sentences. This is consistent with the results in Table 2 that students' scores on the basic Japanese test improved significantly.
Group 4
Scene 1
G4-S1:最近読んだ本は何ですか?(What books have you read recently?)
G4-S2:あのう、これです. <仓央嘉措诗传全集>(Um, here it is. < The Complete Poetic Biography of Kurama Gyatso >)
G4-S1:この本はどうだと思いますか?(What do you think of this book?)
G4-S2:いいです。でも、ちょっと難しい. (It's good. But a little difficult.)
Scene 2
G4-S2: What is your title?
G4-S1: My title is Japanese people.
(Writing and painting)
G4-S3: What are the characteristics of Japanese people?
(Writing and painting)
G4-S2: What’s your title?
G4-S3: I wrote “Japan in Green.”
G4-S2: Oh, OK. Pretty good.
Group 4 had a total of 195 conversations over the two learning activities. There were six conversations about content and 14 conversations about Japanese. Although the overall amount of conversations about content and Japanese in Group 4 was less than in the other three groups, members of Group 4 would try to have conversations in Japanese. In this CLIL class, considering the students' Japanese language level, they were not required to use Japanese to communicate in the group discussions. However, the students in Group 4 would still try to communicate in Japanese independently. This could explain the results in Table 2 that the students' Japanese language scores improved in a meaningful way.
From the conversations in the four groups, it can be seen that the students did not spend all their time discussing the knowledge content and language learning during the application time. Some of the conversations were about how to fill out the worksheet and confirm information about the worksheet. Although the results in Tables 2 and 3 show a significant improvement in Japanese language scores and course content scores, it is also clear from the group discussions that students spent limited time on course content discussions and the use of Japanese. Furthermore, the use of Japanese during group discussions was mostly focused on the confirmation of the problem topics and Japanese words. Only the members of Group 4 tried to communicate entirely in Japanese during the discussion. Therefore, in future CLIL design, attention should also be paid to supporting group discussions. Make group members more active and interactive. Allow students to discuss the course content more.
Result and discussion of research question three
To examine the relationship between designed FPI elements and learning outcomes through CLIL, the correlations between the pre-post difference in learning outcomes, including basic Japanese proficiency test score, content knowledge test score, writing skills test score, and students’ FPI awareness questionnaires, were analyzed.
Table 6 Correlations between FPI, Writing Test, and Basic Japanese Proficiency Test Score
|
Problem-Centered
|
Activation
|
Demonstration
|
Application
|
Integration
|
Basic
|
.554*
|
.605*
|
.520*
|
.387
|
.641**
|
Content
|
.602*
|
.278
|
.303
|
.761**
|
.581*
|
Writing
|
.813**
|
.131
|
.193
|
.675**
|
.624**
|
Note. * p < 0.05; ** p < 0.01.
According to the correlation results in Table 6, some elements of FPI awareness reached a significant level of correlation with learning outcomes.
Correlations among the variables are presented in Table 6. The problem-centered element has a moderate positive correlation between basic Japanese proficiency test score (.554**, p < 0.01), a strong positive correlation between content test score (.602*, p < 0.05), and a strong positive correlation between writing test score (.813**, p < 0.01). As described in the course design phase, this course prepared a life-related problem for each class and had student peers learn the course content to solve the problem. The significant positive correlation that emerged between the problem-centered elements and the learning outcomes in this course is consistent with the findings noted by Gardner (2011). Winch (2016) also points out that classroom activities with clear themes would positively impact learners’ grades.
The activation element has a strong positive correlation between basic Japanese proficiency test scores (.605*, p < 0.05) and no significant correlation between content test scores and writing test scores. It is presumed that there is no obvious storyline that appears in each lesson throughout the course, which may be the cause of this result. For example, the problem in the first lesson was, how do you and your friends self-explain to each other? The problem in the second lesson is, what is your impression of Japanese people in real life, and how can you make better friends with foreigners? Such a question setting may cause students to have a weak sense of the connection between the lessons. They cannot clearly appreciate the connection between what they learned in the previous and the current lessons. Merrill (2020) states that courses should be designed so that students solve problems relevant to real life and have strong connections between problems to evoke better what students have already learned and what they have experienced. The problem in the second lesson should be changed as, looking back at the results of your self-explain, what can you do to become better friends with Japanese students? Tying the problems of each lesson into a complete storyline might better mobilize what students have already learned.
The demonstration element has a moderate positive correlation between basic Japanese proficiency test scores (.520*, p < 0.05) but no significant correlation between content test score and writing test score. Although Merrill (2002) demonstrates that examples of FPI (use of media, worksheets, guidance, etc.) can make more effective use of the lesson by rationally utilizing these elements that students are used to using in other scenarios, the task of effectively integrating multimedia with content and activities is still challenging. Lo and Hew (2018) point out that students can only learn relatively simple knowledge by watching multimedia videos, and they still need guidance and help from teachers when learning complex knowledge. At present, using multimedia in class has become a common trend, but the efficient use of multimedia is still a rather urgent problem to be discussed and solved.
The application element has a strong positive correlation between content test scores (.761**, p < 0.01) and a strong positive correlation between writing test scores (.675**, p < 0.01), but no significant correlation between basic Japanese proficiency test scores. When designing the course, it was decided that there would be a strong relationship between the FPI’s application element and the learning outcomes. As a result, although there were robust correlations between application element and content knowledge, writing skills, but no significant correlation between basic Japanese proficiency. Although the teacher used Japanese when teaching the content and when the students presented their results, they also used Japanese. The analysis of the conversations in the student group discussions showed that the students used their native language most time. When they were in a group discussion, they only used Japanese to confirm topic problems and write the speeches. During the group discussion activities, a total of 1567 conversations took place. However, only 155 (9.89%) conversations occurred in Japanese.
Example 3. Group discussion about the Japanese language.
Group 1
G1-S3: The formation of impressions …
G1-S4: Formative from ドラマ (Drama).
G1-S1: I'm better at writing that way.
(Writing and painting)
Group 1 had a total of 443 conversations in 2 learning activities. There were 30 (6.77%) conversations that included Japanese. As this example shows, in the second learning activity, out of 124 conversations in the group discussion, only five appeared Japanese, and all of them were only Japanese words. This explains the results in Table 6 that the application elements only showed significant correlations with scores on the Japanese writing test and no significant correlations with scores on the Basic Japanese test. Students spent most of their time in group discussions conceptualizing the composition of their writing rather than studying Japanese.
Group 2
G2-S2:Eh, how should 厳しい(strict) be written?
G2-S3:seriousness,巌粛(solemnly)
G2-S1:You can't write 巌粛(solemnly), 巌粛(solemnly) means the atmosphere is serious.
G2-S2:OK, then write厳しい(strict)
20s later
G2-S3:I'll write another one, はじめに(firstly), ...... this one can be written in third.
G2-S2:厳しい(strict)
5s later
G2-S3:OK, what else?
G2-S4:根性 (guts)
G2-S3:根性強い (Strong guts). It's okay, right?
G2-S4:Yeah.
Group 2 had 583 conversations in 2 learning activities, including 77 conversations in Japanese (13.21%). Same as in Group 1, Japanese words appear in most conversations. Nevertheless, as the example shows, a characteristic of Group 2 is that the students think about the deeper meanings of words, and the group members discuss together the appropriateness of the words used. This explains the results presented in Table 6. While no significant correlation was shown between the application element and the basic Japanese test scores, a significantly strong correlation showed between the application element and the writing score. When Group 2 students discussed, they did not come in terms of the basic words taught but in terms of what words they could use to better express their ideas in writing.
Group 3
G3-S2: Hey, Japanese people who love to give gifts, in Japanese it should be that “プレゼントを上げるのがすきな日本人”
G3-S3: Japanese people who like to give gifts, right? プレゼントを贈ることがすき. Like to send gifts, right?
G3-S2: Right, right.
(Writing and painting)
G3-S4: What is this theme of yours?
G3-S2: My theme is Japanese people who like to give gifts.
G3-S3: What about specific? Specific.
G3-S2: Do I say it in Chinese?
G3-S3: OK.
G3-S2: It was that at first, I just thought that Japanese people were giving to Japanese people, but then I found out that they invited people from other countries too, and then my impression of them changed, and I thought they were warm. Then every time we met, they always sent me a gift.
Group 3 had 347 conversations in 2 learning activities, and 34 conversations included Japanese (9.80%). Most of the dialogues also contain only Japanese words. However, it is worth noting that, as the example shows, the members of Group 3 correct each other's incorrect grammar usage. The grammatical use they corrected was not what was taught in this course. This explains the significantly stronger correlation in Table 6 between the integration elements and the Basic Japanese scores. For such grammar correction, it is not only effective for this learning only, but students will think about the situation that was corrected this time and then use grammar correctly when they use Japanese in the future.
Group 4
G4-S2: This write the 環境守る(Protecting the environment) or write the 方法 (method).
G4-S3: En, 方法 (method).
(10 seconds)
G4-S2: 保護 (Protection)
G4-S3: Yes, the pronunciation isほご (Protection).
(Writing and painting)
Group 4 had 195 conversations in 2 learning activities, including 14 conversations in Japanese (7.18%). As in the other three groups, most of the Japanese that appeared in the conversations in Group 4 were words. This is consistent with the results shown in Table 6. Although the usage and pronunciation of words were discussed during the activities, overall, the use of Japanese was least frequent in Group 4.
As the group discussions example showed, it can be seen that the students use their first language most of the time. Moreover, almost all of them appeared as Japanese words, not as fully expressed sentences. Therefore, students actually use Japanese less than other FPI elements during the course time that belongs to the application element. Macaro, Tian, and Chu (2020) point out that when students learn complex knowledge, using their foreign Language for discussion may cause a heavy cognitive burden. Therefore, this course is designed to allow students to use their native language in their discussions. However, Lo (2015) also points out that excessive use of a native language may result in certain obstacles to learners' target language learning. Thus, in future course design, the use of native and foreign languages should be appropriately balanced. In addition, when using FPI for CLIL design, it is necessary not only to ensure that students have enough time for the learning activities but also to improve the quality of the activities.
The integration element has a strong positive correlation between basic Japanese proficiency test score (.641**, p < 0.01), a moderate positive correlation between content test score (.581*, p < 0.05), and a strong positive correlation between writing test score (.624**, p < 0.01). The correlation results are consistent with the questionnaire results about FPI awareness. Students had a high awareness of the integration elements in the course, and this awareness had a positive impact on students' basic Japanese language knowledge, writing skills, and content knowledge. Merrill (2013) points out that the integration element includes reflection, watch me, and creation. As described in the course design, this course gave students the opportunity to reflect on themselves through group discussions, present what they had learned, and think about how they could creatively apply what they had learned in their future lives. The results show that all these designs have had a positive impact on students' learning outcomes.