3.2 Limitations and solutions
In addition to many advantages, the MEEGA + scale also presents some flaws. First of all, we found that three items (”The software allows customizing the appearance (font and/or color) according to my preferences”, ”The software prevents me from making mistakes”, ”There was something interesting at the beginning of the game that captured my attention”) were not really appropriate to evaluate player experience or usability. In the first case, we think that customization of appearance has little to do with the accessibility of the game, dimension to which the item refers. As for the second item, trial and error is an important method in problem solving [Campbell, 1960] and we think that a game that prevents the player from making mistakes is not necessarily more effective than games that let the player progress by trial and error. Finally, we think that discovering something interesting at the beginning does not guarantee that interest will be maintained throughout the game. Moreover, even if you do not find any interesting feature right at the beginning of the game, that does not preclude you from finding it interesting later on. We decided to remove these items in the MEEGA360 scale.
Another problem with the scale is that Petri and colleagues [Petri et al., 2018] note that MEEGA [Savi et al., 2011] - the scale from which MEEGA + is derived - assesses motivation, learning, and user experience in educational games. However, motivation is not directly assessed by the MEEGA + scale. Fun, immersion, and satisfaction of the needs for autonomy, competence, and relatedness underlie motivation but do not overlap with it. We considered adding new items to specifically assess motivation and engagement in the educational game. Since the total of the scale is given by summing the scores of the individual items, we thought that changing the total number of items might change the validity and reliability of the scale in a consistent way. Therefore, we decided to replace the three items described above with three new items related to motivation (”I was motivated more by the game itself than by external rewards”, ”I was determined to finish the game”) and engagement (”I was deeply concentrated in the game”). The MEEGA + scale does not contain any reverse statement. The issue of including or excluding reverse items is complicated, and many reasons can be found in the literature for both using and avoiding reverse items (see Vigil-Cole and colleagues [Vigil Colet et al., 2020] for a full dissertation on the topic). Reverse items are proposed primarily to control for acquiescence bias, but this practice can affect reliability [Krosnick and Presser, 2010]. Acquiescence bias is more common among individuals with lower social status [Lenski and Leggett, 1960, Krosnick and Presser, 2010] and fewer years of formal education [Narayan and Krosnick, 1996, Krosnick and Presser, 2010], and because educational games are often intended for students, it may be useful to control for acquiescence bias with reverse items. To balance the advantages and disadvantages of controlling for acquiescence bias while not compromising reliability, we chose to include four reverse items (”I find this game complicated”, ”I find this game annoying”, ”The game contents are distant to my interests”, ”The game becomes monotonous”) without including them in the evaluation score. In addition, we chose to phrase the items in such a way that they change the direction of the overall meaning of the sentence without using negations, because it is well known that including negations in a questionnaire can cause confusion in the reader and increase the possibility of errors [Wason, 1961]. Furthermore, reverse items are effective when the strength of the statement matches the meaning of the original, positive one, and this cannot be achieved by simply adding a negation in the sentence [Krosnick and Presser, 2010].
Finally, in Petri and colleagues’ scale [Petri et al., 2018], participants express their agreement with each item through a five-point Likert-type scale. As is so often the case with Likert-type scales, the edges of the MEEGA + are excessively distant from the center (i.e., from strongly disagree to strongly agree). Though, this common practice may lead participants to be biased to select items closer to the center rather than the distant ones [Alexandrov, 2010, Samuelson and Zeckhauser, 1988]. Also, verbal anchors tend to influence the perceived distance between the points of the scale to which they refer [Lantz, 2013] and this is the basis for a perceived asymmetry in the scale that causes an end of scale effect. Therefore, we decided to use more moderate verbal anchors, especially for the edges of the scale (disagree, somewhat disagree, neither agree or disagree, somewhat agree, agree).