1. Descriptive statistics
Table 3 presents the averages and standard deviations for each of the six expected scores (three midterm and three final), the total difference scores, the averages and standard deviations of the actual midterm and final exam scores, and the strategy use scores. The difference score for expected final exam score indicated markedly lower expected final exam scores than expected midterm exam scores.
Table 4 shows the descriptive statistics for the learning strategy use scores and the rankings for the utility of the learning strategy according to strategy use priority. Following the above-described scoring pattern of 5 points for low-utility strategies, 3 points for strategies of moderate utility, and 1 point for high-utility strategies, we calculated an average use score of 4.62 for the 105 learning strategies that respondents selected as the first priority; this average was near the 5 points that we assigned to low-utility strategies. The average score for the 105 strategies that the respondent selected as the third priority was 3.88, which is relatively close to our score of 3 points for strategies of moderate utility.
Expected and actual midterm and final exam scores were examined for above-average effects (i.e., above-average effect, better-than-average effect, comparative bias, positive illusions, self-enhancement effect) in differences (Davidai & Deri, 2019; Heck, Simons, & Chabris, 2018; Moore, 2007; Williams & Gilovich, 2008; Zell, Strickhouser, Sedikides, & Alicke, 2020) to determine whether the average expected scores were higher or lower than the actual scores. Table 5 shows that for the midterm exam, more than 99% of students expected higher scores than they achieved and the ratio of over- to underestimating performance was 66:34 for the final exam scores. The reason for this may be that the students’ knowledge of their midterm scores affected their predictions for their final scores.
2. Correlations between the difference score and the actual score
We calculated significant correlations between the expected and actual midterm scores and the expected and actual final exam scores as -0.685 and -0.609, respectively (p < 0.01). Confirming the Dunning-Kruger effect, students who expected high scores generally scored poorly, and students who predicted low scores earned relatively high scores. Table 6 presents actual midterm and final exam score ranges according to score quartiles. As the table shows, the midterm exam score range for the high-performing group was larger than the ranges for the other quartiles, and the ranges for the high- and low-performing quartiles for the final exam were markedly different from the ranges for the other two quartiles.
Table 6. Ranges of actual midterm and final exam scores by quartile.
|
Midterm exam
|
Quartile
|
Final exam
|
Max.–min. score
|
Range
|
# of cases
|
Max.–min. score
|
Range
|
# of cases
|
73–46
|
27
|
26
|
High quartile
|
89.0–64.5
|
24.5
|
26
|
46–34
|
12
|
26
|
3rd quartile
|
64.5–49.5
|
15.0
|
26
|
33–21
|
12
|
26
|
2nd quartile
|
49.0–33.5
|
15.5
|
26
|
20–7
|
13
|
27
|
Low quartile
|
33.5–8.0
|
25.5
|
27
|
Figure 2 presents the trends for differences between expected and actual midterm and final exam scores by quartile. It was found that the lowest-performing group showed the greatest overestimations of expected test score. The lower-achieving students also greatly overestimated their predicted final exam scores, and the higher-performing students anticipated lower final exam scores. In other words, on both the midterm and final exam scores, the students in this study manifested the Dunning-Kruger effect.
After the actual midterm and final exam scores were divided into quartiles, the differences between each expected score and the actual score were averaged, and Figures 3 to 5 present these change trends. Figure 3 shows that the maximum midterm exam difference score was 56.78 points (low quartile, 1st difference score), and the minimum was 21.77 points (high quartile, 3rd difference score). Figure 4 reveals that the maximum final exam difference score was 22.89 points (low quartile, 5th difference score), and the minimum was 11.79 points (3rd quartile, 4th difference score). Figure 5 indicates that the final exam difference score was three times lower than that for the midterm exam (35.01 vs. 11.10 points).
Figure 6 shows the distributions of differences between expected and actual midterm and final exam scores. The figure demonstrates that the high-performing quintile exhibited relatively high difference scores for the final exam, i.e., many students in this quartile underestimated their expected scores.
3. Difference in estimated scores before and after the midterm exam
1) Correlation and mean difference test
Figure 5 shows that the students in this study generally predicted significantly higher midterm exam scores than final exam scores. Table 7 also reveals that the correlations were very high between the three expected midterm exam scores and between the three expected final exam scores.
Table 7. Correlations between expected and actual midterm and final exam scores.
|
|
Final actual score
|
Expected score
|
Midterm exam
|
Final exam
|
1st trial
|
2nd trial
|
3rd trial
|
4th trial
|
5th trial
|
6th trial
|
Actual score
|
Midterm exam
|
.831**
|
.181
|
.224*
|
.251**
|
.416**
|
.350**
|
.304**
|
Final exam
|
|
.118
|
.222*
|
.245*
|
.425**
|
.384**
|
.376**
|
Difference score
|
Midterm exam
|
1st trial
|
|
|
.777**
|
.679**
|
.466**
|
.510**
|
.467**
|
2nd trial
|
|
|
|
.783**
|
.521**
|
.567**
|
.534**
|
3rd trial
|
|
|
|
|
.554**
|
.584**
|
.503**
|
Final exam
|
4th trial
|
|
|
|
|
|
.924**
|
.835**
|
5th trial
|
|
|
|
|
|
|
.895**
|
*p < .05, **p < .01.
|
Larger difference scores indicate less-accurate estimates of actual performance, so that the difference score and the actual score will exhibit a negative correlation. Table 8 shows that this negative correlation was higher for the midterm than for the final exams.
Table 8. Correlations between midterm and final exam difference and actual scores.
|
|
Final actual score
|
Difference score
|
Midterm exam
|
Final exam
|
1st trial
|
2nd trial
|
3rd trial
|
4th trial
|
5th trial
|
6th trial
|
Actual score
|
Midterm exam
|
.831**
|
-.689**
|
-.679**
|
-.600**
|
-.190
|
-.240*
|
-.194*
|
Final exam
|
|
-.595**
|
-.538**
|
-.470**
|
-.212*
|
-.229*
|
-.127
|
Difference score
|
Midterm exam
|
1st trial
|
|
|
.887**
|
.815**
|
.240*
|
.270**
|
.185
|
2nd trial
|
|
|
|
.863**
|
.207*
|
.259**
|
.273**
|
3rd trial
|
|
|
|
|
.287**
|
.365**
|
.339**
|
Final exam
|
4th trial
|
|
|
|
|
|
.887**
|
.755**
|
5th trial
|
|
|
|
|
|
|
.825**
|
**p < .01, *p < .05.
|
When we calculated the correlations between the actual scores and each of the three expected scores and divided those correlations into quartiles, high positive correlations were identified between the three expected scores in each quartile. This finding indicated consistency between score estimates, irrespective of the student’s actual achievement. In fact, both the lowest- and highest-performing students maintained high positive correlations between their three estimated scores.
Six separate mean differences were calculated for this study: the difference between each of a student’s three estimated midterm scores and three estimated final exam scores and that student’s actual midterm and final exam scores. From these calculations, it was shown that the difference between the midterm three-score difference scores and the final exam three-score difference scores was not statistically significant. Overall, however, the students showed lower expected scores for their final exams than they had predicted for their midterm exams, which may be reasonably ascribed to their tempered expectations after they had seen their actual midterm scores. Table 9 shows the results for repeated-measures analysis of variance of each of the six difference scores (F(5,100) = 64.54, p < .001) and Scheffe’s post hoc test results. The table indicates a significant difference between the midterm and final exam score difference scores.
Table 9. Multiple mean differences by number of difference scores.
|
Difference score
|
# of cases
|
Mean
|
SD
|
F
|
p
|
Post hoc
|
Predicted 1st – midterm actual (a)
|
105
|
41.46
|
19.25
|
64.54
|
.000***
|
a, b, c > d, e, f
|
Predicted 2nd - midterm actual (b)
|
41.92
|
18.67
|
Predicted 3rd - midterm actual (c)
|
36.96
|
19.60
|
Predicted 4th - final actual (d)
|
15.83
|
12.80
|
Predicted 5th - final actual (e)
|
17.26
|
12.96
|
Predicted 6th - final actual (f)
|
17.46
|
13.50
|
***p < .001
|
2) Expected final exam score by midterm difference score quartile
When we compared the midterm difference scores divided by quartile with the final exam difference scores divided by quartile, it was shown that students whose midterm difference scores were in the lowest quartile predicted their final exam performance relatively accurately, and the students with difference scores in the highest quartile predicted their final exam scores the most inaccurately. For instance, there were 26 students in the lowest-performing quartile for the midterm exam difference score, and of these, 8 had final exam scores in the lowest quartile; 1 of those students had estimated a higher than actual final exam score, and the other 7 underestimated their final exam scores.
In contrast, students with final exam scores in the highest quartile who had the largest difference between the midterm difference score and the actual score had still far overestimated their predicted-actual final exam scores. Specifically, among the 27 students whose midterm score difference was in the high quartile, 12 had final scores that were still in the high quartile, but all 27 had anticipated higher scores than their actual final scores. Figure 7 shows the students’ final exam difference scores according to their midterm exam difference score quartiles. Overall, the lower was the quartile of the difference score, the more accurate was the score prediction.
Table 10 presents detailed support for the Dunning-Kruger effect in this study’s results, as evident in the differences between the students’ actual test performance (competence) and what they believed their competence to be (their illusions). The 26 students with midterm exam difference scores in the lowest quartile showed the smallest differences between their expected and actual midterm exam scores, and thus can be considered the most competent at assessing their own skills. Table 10 shows that 23 students in that quartile estimated lower midterm exam scores than they earned, and only 3 who had overestimated their scores had underestimated their predicted final exam scores. In contrast, the 27 students with midterm exam difference scores in the highest quartile, i.e., who showed the largest differences between expected and actual midterm scores, can be considered incompetent at judging their competence; only 2 underestimated their actual midterm exam scores, and 25 still overestimated their actual midterm exam scores. The findings shown in the table clearly demonstrate the bidirectional nature of the Dunning-Kruger effect
Figure 8 presents the difference between the midterm and the final exam difference score for each student in each quartile. Overall, the slope of each straight line is downward to the right, which indicates that the difference score for the final exam is smaller than the difference score for the midterm exam. It is concluded that this finding confirms our proposal that students more accurately predicted their final exam scores because they had evidence of their midterm exam scores. Table 11 shows that students with low difference scores by quartile tended to greatly overestimate their predicted scores, and students in the highest quartile for difference scores continued to underestimate their predicted scores.
Figure 9 shows the sum of the three expected scores for the midterm exam and the actual score difference, i.e., the student with the lowest midterm difference score to the student with the largest midterm difference score, and then displays the final exam difference score of each student. As shown in the figure, where all midterm difference scores were positive values greater than 0, all students overestimated their competencies, but the final exam difference scores were divided into positive and negative values. In other words, the larger was the midterm difference score, the more the final exam difference score tended to be overestimated, but the lower was the midterm difference score, the more it was underestimated, indicating that most cases were negative values less than 0. Through the comparison of the difference scores between the midterm and final exams, it can be seen that the performance prediction for the midterm exam was highly overestimated, and the bidirectional nature of the Dunning-Kruger effect was clearly confirmed in the final exam performance estimation. Moreover, the foresight bias phenomenon that occurred in the midterm grade estimation process was significantly resolved in the final grade estimation through an educational prescription called grade confirmation, but the bidirectional relationship of the Dunning-Kruger effect intensified.
3) The change in accuracy of prediction: difference scores
The bidirectional phenomenon of the Dunning-Kruger effect can be confirmed by differences changes between the actual midterm and final exam scores and their difference scores. Figure 10 shows the frequency of midterm difference scores by grade according to the actual midterm score quartile. The difference score range was 0 to 240, and we separated the range into 12 grades of 20 points each. Students with high actual midterm exam scores were distributed at the low end of the difference score spectrum, and students with low actual scores were distributed at the high end of the spectrum.
The distribution presented in Figure 10 changed significantly following the final exam. Figure 11 shows the frequencies of the final exam difference scores by grade and actual final exam score quartile. Overall, the students with low actual scores and high difference scores shifted to low difference scores for their final exam grades, regardless of the quartile of their actual final exam scores.
To determine how the midterm and final exam actual scores and difference scores changed, we calculated each frequency by converting the 12th difference score grade to the 4th grade, as shown in Table 11. In the lowest quartile for the difference scores, only the top 16 of the 105 scores (15%) were midterm exam scores; scores for 73 students (70%) were final exam scores, regardless of the score. In addition, all 17 students in the highest quartile (largest difference between expected and actual midterm score) and 19 students of the 29 (66%) in the 3rd quartile had among the lowest actual scores. These findings verified that students’ knowledge of their actual midterm scores had made their estimates of their final exam scores more reliable.
4. Learning strategy use, actual scores, and difference scores
Table 12 presents the results of comparing the learning strategies that students used with their actual and difference scores for the midterm and final exams. The correlation between the total score (midterm exam score + final exam score) and the learning strategies used is that students with high achievement used relatively less-useful strategies than did students with lower scores. Furthermore, this trend held for both midterm and final exams and the strategies used.
Table 12. Relationships between learning strategies used, difference scores, and actual scores.
|
Examination
|
Comparison
|
Correlation
|
Midterm + final
|
Actual score vs. strategies used
|
−.386**
|
Midterm exam
|
Actual score vs. strategies used
|
−.368**
|
Final exam
|
Actual score vs. strategies used
|
−.352**
|
Midterm exam
|
Difference score vs. strategies used
|
.251**
|
Final exam
|
Difference score vs. strategies used
|
.238*
|
*p < .05, **p < .01.
|
In contrast with the actual score relationship, difference score had positive correlations with learning strategies used for both midterm and final exam scores. In other words, when the difference score was high, students tended to use more learning strategies, but students with higher difference scores showed less-accurate predictions of their performance. Therefore, use of more learning strategies reflects that students who lack the ability to objectively judge their competence used less-useful learning strategies. Figures 12 and 13 show that in the relationships between learning strategies used and the actual difference scores for the midterm exam and the final exam, respectively, students tended to use more learning strategies as their actual scores increased and their difference scores decreased as their actual scores increased, especially in the top grades.
5. Effects of difference scores and learning strategies used on actual scores
The results of multiple regression analysis of the effects of difference scores and learning strategies used on actual midterm and final exam scores are as follows: First, each regression model was statistically significant (midterm exam F(2,102) = 58.422, p < .001; final exam F(2,102) = 7.262, p < .001) The explanatory power of the regression model was 53.4% for the midterm exam scores and 12.5% for the final exam scores. The Durbin-Watson value was 2.068 for midterm scores and 1.571 for final scores, both of which were near 2, which indicated that the assumption of independence of residuals was valid. We calculated correlations between variables of R = .731 for the midterm exam and R = .353 for the final exam, and tolerance and VIF were 0.1 or more and less than 10, respectively; in both cases, no multicollinearity problem was present.
In calculating the significance of the regression coefficient, excluding the difference score for the final exam (p = .123), the midterm exam difference score (p < .001) and learning strategies used for the two exams (p < .01) had significant negative effects on each actual score. It was found that the actual score decreased as the difference score and the strategy prioritizations increased. When we checked the non-standardization coefficients for the significant variables, the midterm exam difference score (B = -.185) and learning strategies used (B = -.880) and the final exam difference score (B = -.079) and learning strategies used (B = -1.315) were all negative. Actual midterm and final exam scores decreased as the difference score and learning strategy utility increased, and the standardization coefficients were as follows: midterm exam difference score, = −.639; midterm learning strategies used, = −.247; final exam difference score, = −.147; and final exam learning strategies used, = −.296. This study verified that the learning strategies used had a more significant influence on the actual midterm score than the midterm difference score. Additionally, only learning strategies used affected the actual score because we did not secure the difference scores for the final exam. The regression equation representing the degree of increase in midterm exam score according to difference score (X1) and learning strategies used (X2) was , and the regression equation representing the degree of increase in final exam score was
In terms of the learning strategies that the students reported using, they each selected the three that they used most frequently, and we prioritized them and assigned points according to priorities (5 points for strategies of low utility, 3 points for strategies of moderate utility, and 1 point for strategies of high utility). The students scored the first- and second-priority strategies 4.62 and 4.49, respectively, which were close to the 5 points for low-utility strategies; the third strategy scored 3.88 points, between the moderate and high strategies. Overall, the students overwhelmingly chose learning strategies of low utility at 76.8%, followed by strategies of moderate and high utility at 12.7% and 10.5%, respectively, which was in accordance with findings from relevant literature (Kang, 2017; Karpicke, 2017; Seok & Kang, 2019). In addition, the correlation coefficient between the learning strategy utility and the total score and the midterm and final test scores were all significantly negative at .001. A negative correlation between the test score and the learning strategy indicated that the utility of the learning strategy used had an inverse relationship with the actual exam test score. This result reflects that students with high academic achievement used learning strategies with more utility, which supported previous studies’ findings that high-utility learning strategies have greater learning effects than do strategies of moderate or low utility.