The data gathered from the pre- and post-tests were then subjected to statistical analysis. The Wilcoxon Signed Ranked Test was used to determine whether there was a significant difference between the two test scores and compare the results of the students’ use of DMs in the two with the goal to determine whether using GBA can serve to provide practical and effective guidance in developing competent English speakers by raising their awareness of discourse competence when reading different genres to identify DMs. Secondly, data analysis for the pre- and post-test questionnaire was conducted by computing descriptive statistics to determine the students’ knowledge of DMs before and after the intervention. Thirdly, the data from the interviews with the students responding to the nine questions were transcribed and analyzed to identify their experience of learning the DMs in this class.
4.1. Results
RQ1: To what extent do the university students know about DMs before the teaching intervention?
To determine the students’ understanding of the oral assessment, seven questions were asked. One was answered on a Likert Scale to determine the frequency of the students’ experiences in speaking English in class in senior high school, three yes-no questions were used to determine the students’ experiences in being evaluated when speaking English, and three open-ended questions assessed their knowledge of the oral assessment’s elements.
First, the results indicated that 50% of the students had opportunities to speak English in class when they were in senior high school, while 35.7% of them seldom, and 14.3% of them never. Second, 53.6% of them had received feedback on their speaking skills, while 46.4% of them never had. Thirdly, the results showed that 35.7% of them knew the way speaking skills were evaluated, while 64.3% of them indicated that they lacked such understanding. Fourth, with respect to the percentage of the students who took an official oral proficiency test, 28.6% of them had taken such test, and 71.4% of them answered no for this experience.
To understand the students’ knowledge of an oral proficiency test’s criteria further, Questions 5, 6, and 7 were open-ended for them to offer their answers in words. The answers to each question are listed below. The students’ knowledge of the elements assessed in an oral proficiency test is presented in Fig. 3 showing that the students know the elements evaluated during an oral proficiency test, such as fluency, vocabulary, grammar, and pronunciation, etc. Among the four main elements, fluency was the answer provided the most, in which 13 of the 28 (46.4%) recognized the significance of fluency in a speaking test. Pronunciation, to which 12 (42.9%) referred, and was followed by grammar and vocabulary, mentioned by 9 (32.1%).
The students’ responses to the way they were informed about the elements in the oral assessment are presented in Fig. 4, indicating that 9 (32.1%) learned the elements assessed in an oral test from teachers, 8 (28.6%) through their own experience, 14.3% from test-related information on the Internet, and 3 (10.7%) not having access to such information.
The students’ responses with respect to their opinion of the most important competences in an oral proficiency test were calculated and are presented in Fig. 5. This indicated that the competences for an oral test were as follows: 6 (21.4%) mentioned accuracy in grammar, 5 (17.9%) vocabulary, 4 (14.3%) pronunciation, 2 (7.1%) fluency, and there were the remaining items that were not evaluated during an oral proficiency test.
RQ2: To what extent can GBA serve the function of providing practical and effective guidance in developing competent English speakers by raising students’ awareness of discourse competence through reading and listening in different genres for discourse markers?
The students’ performances were evaluated according to the criteria of the Speaking Feedback Sheet adapted from IELTS Academic (2016) comprised of four parts: Fluency and coherence; lexical resource; grammatical range and accuracy, and pronunciation.
Each student’s total speaking score was calculated by summing the four aspects of speaking with three sub-aspects for each. The students were given scores ranging from 0 to 3 for each, as follows: 0 = Not all; 1 = A little; 2 = Sometimes, 3 = Always. Then the total score was divided by 4. Hence, if a student had a total score of 36 (12 for each aspect), his/her score for speaking was 9 (36 divided by 4).
The results of the pre-test shown in Table 1 indicated that the mean of the pre-test scores was 10.93. Compared to the total score for the four speaking criteria, 36, the mean of the pre-test scores showed that initially, the students had difficulty speaking English and had no good command of using DMs for cohesion and coherence in the speech.
The post-test was conducted to determine the students’ speaking proficiency and their use of DMs after the teaching intervention. The results of the post-test indicated the mean score was 18.43 (see Appendix A). Compared to the total scores of the pre-test, 10.93, the mean post-test score showed that the students’ speaking proficiency improved.
The result of the Wilcoxon signed rank test indicated a significant difference (Z = -4.409, p = 0.000) between the pre- and post-test scores. The median score for the post-test after the teaching intervention was 18 compared to 11 before the GBA.
The effect size was also calculated by dividing the absolute (positive) Standardized test statistic z by the square root of the number of pairs. r = z (-4.409) / √56 = 0.59. The effect size, 0.59, was very large according to Cohen’s classification of effect sizes, which is 0.1 (small effect), 0.3 (moderate effect), and 0.5 and above (large effect).
Rather than asking what the students knew about DMs directly, the frequencies of DMs in the pre-test were recorded, transcribed, and calculated, and are presented in Table 1 with the goal to observe their use of DMs in their one-minute speech before the intervention. As it shows, the total number of DMs used in the pre-test was 195, with a mean of 6.96. The minimum number of DMs used was 2, while the maximum was 12.
To determine whether GBA offered practical and effective guidance in developing the students’ competence in speaking English by raising their awareness of discourse competence through reading different genres to identify DMs, the frequencies of DMs the students applied in the post-test were calculated and are presented in Table 1; further, the DMs were transcribed to see the changes that the GBA may have brought about. As it shows, the total number of DMs used in the post-test was 244, with a mean of 8.71, a minimum of 2, and a maximum of 16.
The comparison of the students’ use of DMs in the pre- and post-test was conducted using the Wilcoxon signed-rank test, which showed that the 13-week, twice weekly intervention produced a statistically significant change in the 28 students’ use of DMs (Z = -2.107, p = 0.035). The median score for the post-test after the teaching intervention was 9 compared to 7 before the GBA.
Table 1.
The number of DMs used in the pre- and post-test
|
The total No. of DMs used in
|
Student
|
the pre-test
|
the post-test
|
1–5
|
12, 5, 9, 7, 12
|
14, 11, 11, 11, 7
|
6–10
|
8, 2, 12, 2, 7
|
10, 4, 9, 9, 7
|
11–15
|
5, 8, 9, 4, 9
|
6, 3, 13, 13, 9
|
16–20
|
9,10, 4, 6, 5
|
9, 5, 6, 8, 2
|
21–25
|
6, 7, 5, 3, 7
|
12, 9, 11, 4, 7
|
26–28
|
7, 7, 8
|
11, 7, 16
|
Total No. of DMs
|
195
|
244
|
Mean
|
6.96
|
8.71
|
Min
|
2
|
2
|
Max
|
12
|
16
|
RQ3: To what extent can teaching DMs help enhance students’ ability to structure and manage discourse with respect to coherence and cohesion?
To explore the way the students managed discourse with respect to coherence and cohesion further, a detailed analysis was conducted on their use of DMs in the pre-test. Among the 195 DMs, “and” was used most often, 79 times; “so” the second most, 24 times; “because,” 17 times; “but,” 15 times, and “when,” 15 times. Table 2 shows that in the pre-test, the students used 19 DMs in their speeches, which can be categorized into 8 groups according to its meaning: Addition; cause; effect; contrast; temporal sequence; temporal overlap; condition, and example (See Appendix B).
Table 2.
List of the DMs used in the pre-test
DM1
|
and
|
DM6
|
before
|
DM11
|
for example
|
DM16
|
then
|
DM2
|
after
|
DM7
|
because
|
DM12
|
however
|
DM17
|
the reason why
|
DM3
|
and then
|
DM8
|
due to
|
DM13
|
if
|
DM18
|
the reason is that
|
DM4
|
at that moment
|
DM9
|
every time
|
DM14
|
like
|
DM19
|
when
|
DM5
|
but
|
DM10
|
finally
|
DM15
|
so
|
|
|
Among the 244 DMs used in the post-test, “and” was used most often, 57 times, indicating a decrease compared with the 79 times it was used in the pre-test. “If” was used second most, 42 times. The third most was “but,” 32 times. As is presented in Table 3, 46 DMs appeared in the post-test can be categorized into 12 groups: Addition; cause; effect; comparison; contrast (concession); purpose; temporal sequence; temporal overlap; example; condition; opinion, and summary/conclusion (See Appendix C).
Table 3.
List of the DMs used in the post-test
DM1
|
and
|
DM16
|
even
|
DM31
|
instead of
|
DM2
|
also
|
DM17
|
even if
|
DM32
|
last
|
DM3
|
although
|
DM18
|
even though
|
DM33
|
moreover
|
DM4
|
after
|
DM19
|
first
|
DM34
|
nonetheless
|
DM5
|
and then
|
DM20
|
finally
|
DM35
|
not only… but also
|
DM6
|
as well
|
DM21
|
furthermore
|
DM36
|
on the other hand
|
DM7
|
as a result
|
DM22
|
for example
|
DM37
|
so
|
DM8
|
as long as
|
DM23
|
generally speaking
|
DM38
|
so that
|
DM9
|
as a saying goes
|
DM24
|
however
|
DM39
|
second
|
DM10
|
at this point
|
DM25
|
hence
|
DM40
|
then
|
DM11
|
but
|
DM26
|
if
|
DM41
|
the reason why
|
DM12
|
before
|
DM27
|
in conclusion
|
DM42
|
take… as an example
|
DM13
|
because
|
DM28
|
in addition
|
DM43
|
take… for example
|
DM14
|
consequently
|
DM29
|
in my opinion
|
DM44
|
unfortunately
|
DM15
|
due to
|
DM30
|
in that way
|
DM45
|
when
|
|
|
|
|
DM46
|
when it comes to
|
Comparing the DMs used in the pre- and post-test, here were three types of results in the students’ use of them in their two tests: the use of DMs increased, remained unchanged, and decreased. As Table 4 shows, 18 students’ use of DMs in their speech increased, while 5 decreased, and 5 remained unchanged.
Table 4.
Comparison of each student’s use of DMs in the pre- and post-test
|
Increase of DMs
|
No change
|
Decrease of DMs
|
Student No.
|
1,2,3,4,6,7,9,11,12,14,15,20,21, 23,24,25,26,28
|
10,16,17,19,27
|
5,8,13,18,22
|
Further, to confirm the extent to which the students learned DMs, the common use of “and,” but,” and “so,” were excluded. Then, the DMs used in the pre- and post-test were compared again. As is shown in Table 5, there was a slight change in the three types of results in their use of DMs in the two tests: 21 students enhanced their awareness and variety of DMs in their speech, while 5 decreased and 3 remained unchanged.
Table 5.
Comparison of each student’s use of DMs in the pre- and post-test without “and” “but” and “so”
|
Increase in DMs
|
No change
|
Decrease in DMs
|
Student No.
|
1,2,3,4,6,7,8,9,10,11,12,14,15,20,21,23,24,25,26,27,28
|
13,19,20
|
5,16,17,18,22
|
RQ4: What are the students’ responses to teaching DMs through GBA and transcription in reflective language learning?
The data from the interviews were explored to assess the students’ responses to teaching DMs through GBA. Several questions were asked about the students’ understanding of the criteria for evaluating speaking skills, knowledge of DMs, application of DMs in the four skills of English learning before and after the teaching intervention, and experience transcribing their own oral tests.
First, with respect to the students’ understanding of the criteria for evaluating speaking skills after the one-semester intervention, the analysis showed 21 students (75%) believed that the teaching intervention gave them a much better understanding of the criteria, as each of them was given individual feedback according to the four main aspects.
Secondly, concerning the students’ understanding of DMs, their answers indicated that most of them understood DMs as conjunction or transition words, or sign posts with different aspects of meanings. They could even offer many DMs for different aspects of meaning, demonstrating their basic understanding of the way to use DMs in their speaking. Further, they had begun to recognize the significance of using DMs appropriately in their speaking, as they expressed that they found their frequent use of “and,” “but, and “so” in their speech; they also began to notice the importance of DMs for organizing the structure of the content in a speech, and one student stated that DMs serve as a signal to continue the talk when giving a speech.
Thirdly, when the students were asked about their use of DMs in listening, speaking, reading, and writing before and after the teaching intervention, their opinions differed greatly. Before the instruction, some of the students indicated that they had past experience of being taught about DMs in their listening (8 students, 28%), reading (12, 43%), and writing (14, 50%). For the listening activities, 8 students expressed their understanding of the significance of using DMs when practicing listening but failed to explain exactly the way they used them when they were actually listening. However, the students apparently understood DMs’ functions when reading and writing. As they stated in the interviews, they thought that DMs had helped them predict what would come next when reading an article and could understand the connections in the content easily. When writing, they believed DMs helped them achieve strong cohesion among sentences, as they can be used to organize ideas and increase the writing’s structure. The responses above indicate the students’ past experience of learning DMs in listening, reading, and writing, but in speaking, 26 (93%), indicated that they had never been taught to use DMs when engaged in speaking activities because of the lack of such opportunities to speak English.
After the instruction, 24 students (86%), agreed that DMs enhanced their understanding when engaged in listening activities, as they served as indicators to help them predict, expect, or pay extra attention to what might follow or occur next; moreover, they helped them follow the flow of the thoughts and determine the key points. 26 (93%) held a positive attitude toward identifying DMs when engaged in reading activities for nearly the same reason indicated in the receptive skill above. All students reached a consensus on applying DMs in their writing to reduce their repetition of such transition words as “and,” “but,” and “so,” and to organize the content’s structure; all of them also agree that applying DMs in their speaking for organizing their thoughts and expressing their points clearly helps the listeners understand their viewpoints.
Ultimately, the students’ responses to learning DMs through transcription for reflective language learning in this GBA was also addressed. The students were required to revise their own oral tests by correcting the elements of sentences and practicing adding necessary DMs. Asked whether this transcription for reflective learning helped improve the quality of their speaking, 26 students (93%) held a positive attitude toward this activity. As most of them pointed out, transcribing after each oral test helped them in several ways: 1) Identifying and correcting their own mistakes in vocabulary and grammar; (2) reflecting on the content’s structure and improving its cohesion by adding DMs appropriately, and (3) learning to organize their thoughts.