Using a product goes beyond meeting physical, usage and operational needs. Cultural and social needs, emotions, and inspirations are also part of this context and assessing emotions must be considered and evaluated (Netzer et al. 2018). The study of the psychological aspects and of the impact of an emotional experience that a product or a system can provoke in the user is a determining factor in design (Löbach 2001). In addition, it is important to study emotional connections with regard both to acquiring products and to using them (Norman 2004; Soares 2021; Ge et al. 2021).
It is believed that a great desire and also a great challenge for designers is to develop products that can cause ever stronger, real and lasting emotions in users. With game designers it is no different, especially for those who develop simulators and especially with Virtual Reality. After all, the more immersive an experience is, the greater the emotional change for the user.
In this context, immersion and presence are key elements. According to Rebelo et al. (2012), Oliveira et al (2016), Tori et al. (2018), immersion is the feeling of being inside an environment, not only visually, but also of experiencing other intensities and senses, such as hearing and touch. Moreover, according to the authors, presence corresponds to the ability of a system to effectively detect the user input. Both are related to biological and emotional phenomena.
The use of simulators allows users' reactions to situations that resemble real life in a virtual representation and a specific situation to be investigated. This type of activity and the pieces of equipment for this aim to simulate real user experiences in an immersive, empathetic and participatory way (Salen and Zimmerman 2003; Martin and Hanington 2012; Drachen et al. 2013).
This type of interactive experimentation also occurs in virtual reality applications by using computationally generated digital environments (Jerald 2015). Therefore, a VR application can also be considered a simulation. The purpose of using it, the construction of the virtual environment and the quality of this application can make it possible to give an immersive, semi-immersive and non-immersive experience to users (Rebelo et al. 2012, Tori et al. 2018).
In several studies in which users' emotions were investigated, various types of data collection techniques were identified. In addition to self-report techniques (e.g., questionnaires), equipment has also been used to assess users’ psychophysiological activity (Zhao et al. 2016; Wang et al. 2017).
Psychophysiological investigations can be defined as those that use physiological signals to study psychological phenomena (Li et al. 2015). They can be considered more reliable due to the impossibility of controlling human consciousness, especially in research studies involving the study of emotional responses (Jenkins et al. 2009). This is because physiological components are involved in the unpredictable reaction of the organism which is prompted by emotions (Lelord and Andre 2002).
Objective, precise, and measurable technological responses are required in this investigative scenario. Neuroimaging techniques meet this demand. For example, analyzing how the brain behaves during cognitive activities and when performing tasks. The EEG is considered the best choice for recording information on brain activity due to its characteristics: it is of a non-invasive nature, it emits a continuous signal, it has sensitivity, it captures signals in real-time and it has excellent temporal resolution (Mehta and Parasuraman 2013; Lecoutre et al. 2015; Zhang et al. 2016). Using EEG in experiments with computational systems, especially games, has enabled changes in several frequency bands of the brain activity of players to be identified (Kivikangas et al. 2011; McMahan et al. 2015; Kerous et al. 2017). Virtual Reality is also inserted into this context (Sun et al. 2019; Rogers et al. 2021).
Another technique with excellent potential for evaluating users’ emotional stimuli is digital infrared thermography (Jenkins et al. 2007). By using infrared radiation, temperature and its distribution in a body or object are recorded (Jones 1998). Recognizing emotions by using thermography is a technique that has become practiced by a growing number of researchers over the last two decades (Or and Duffy 2007; Nakanishi and Imai-Matsumura 2008; Barros et al. 2016; Jian et al. 2017).
Although there is no standard model of thermal analysis that enables human emotions to be recognized (Fu and Frasson 2016; Filippini et al. 2020), monitoring the change in temperature of Regions of Interest (ROI) on the face has allowed correlations with specific emotions and two-dimensional models of emotion to be identified (Merla and Romani 2007; Robinson et al. 2012; Ioannou et al. 2014; Salazar-López et al. 2015; Diaz-Piedra et al. 2019).
As for cognitive aspects, thermography has already been used in multimodal studies that have compared self-report responses (what is said) with physiological responses (what is felt) in studies on the veracity of information given by individuals to public authorities in everyday situations e.g., at airports (Pavlidis 2004; Warmelink et al. 2011).
Currently, digital games, with or without virtual reality technology, have become important fields of research on human emotion for several reasons: the need to use mixed methods of investigation given the diversity of elements involved (Lieberoth and Roepstorff 2015), the need for such investigations to require various competencies to be deployed collaboratively (Kampa 2020; Pearson 2020), and because of the potential for using multimodal systems (Mishra et al. 2016; Kotsia et al. 2016).
Electronic games have long met demands that go beyond the desire for entertainment. Serious games offer good examples of this because they have the potential to develop users’ perceptual, cognitive, and motor skills (Connolly et al. 2012; Jerzak and Rebelo 2014). For instance, Boyle et al. (2016), Tahmosybayat et al. (2018) and Neves (2022) draw attention to the successful use of electronic games to improve the quality of life of people with health problems.
Several studies have made use of racing games. This type of game simulates the need for the user to learn and/or improve the driving skills needed in nearly all social contexts related to human mobility. A vehicle driving simulator, whether used for entertainment or training, immerses users in situations with which they empathize because what they experience is real to them (Salen and Zimmerman 2003; Martin and Hanington 2012). Several multimodal studies have already made use of this category of games (Tognetti et al. 2010; Uriarte et al. 2015; Lee and Bae 2019), including studies on virtual reality (Hartfiel and Stark 2021).
In view of this context, the main objective of the present study was to examine whether users' self-reported responses about their emotional state when using driving simulators show any emotional valence correlation with their psychophysiological reactions. By putting users in a fun situation and another one for training purposes, it was possible to check in which one there was a greater correlation between the answers reported in the questionnaires and the psychophysiological responses obtained by the users’ brain signals and the temperature of their faces.