How individual factors intervene in the expression of cognitive abilities in a virtual immersive task: a validation study of the Virtual Reality Action Test

doi:10.21203/rs.3.rs-2598757/v1

Download PDF

Article

How individual factors intervene in the expression of cognitive abilities in a virtual immersive task: a validation study of the Virtual Reality Action Test

https://doi.org/10.21203/rs.3.rs-2598757/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 17 Aug, 2023

Read the published version in Scientific Reports →

You are reading this latest preprint version

There is a raised interest in literature to use Virtual Reality (VR) technology as an assessment tool for cognitive domains. One of the essential advantages of transforming tests in an immersive virtual environment is the possibility of automatically calculating the test's score, a time-consuming process under natural conditions. Although the characteristics of VR can deliver different degrees of immersion in a virtual environment, the sense of presence could jeopardize the evolution of these practices. The sense of presence results from a complex interaction between human factors and the proprieties of VR devices. The present study has two aims: firstly, it contributes to the validation of a virtual version of the Naturalistic Action Test (i.e., Virtual Reality Action Test); second, it aims to evaluate the role of sense of presence as a critical booster of the expression of cognitive abilities during virtual reality tasks. The study relies on healthy adults tested in virtual and real conditions in a cross-over research design. The study's results support the validity of the Virtual Reality Action Test. Furthermore, two structural equation models are tested to comprehend the role of sense of presence as a moderator in the relationship between cognitive abilities and virtual task performance.

Biological sciences/Psychology

Health sciences/Health care

Virtual reality

performance-based test

sense of presence

immersion

cognitive assessment

In recent years, the use of Virtual Reality (VR) technology and immersive interface have attracted the interest of several scholars ^1–3. VR has been traditionally defined as “interactive, virtual image displays enhanced by special processing and by non-visual display modalities […] to convince users that they are immersed in a synthetic space” ⁴. The primary purpose of this technology is to provide an authentic and immersive experience, replacing real stimuli with high realism that can be customized and integrated into an ecological task ^5–7. The simulated environment can be easily manipulated, facilitating experimental tasks that are difficult to implement in real-world settings. Other benefits of VR regard the possibility of participating in potentially dangerous tasks, such as moving in a complex environment or applying psychological treatment to problems arising from phobias⁸ in a controlled ecological setting ⁹. Although VR technology has an interdisciplinary nature, in the last few years, it might appear that most developments in VR studies have focused on clinical aspects ⁵. Nowadays, VR emerges as a promising valuable technology as an effective medium for administering different interventions in the healthcare contexts ^10,11. Moreover, VR is considered a valuable tool that could improve and automatize the processes of administering and scoring traditional performance-based assessments without jeopardizing ecological validity. Different research highlighted that the score calculated from VR devices reduces the number of errors that could distort the results¹² and reduces the time required for the scoring ¹¹. The technological progress of VR has expanded the range of tools and types of research questions, adapting several standard performance-based tests to the virtual version. Different studies applied VR tools to assess executive functions¹³ and spatial abilities ^5,14, showing promising results. However, several performance-based tests are traditionally delivered in real-life settings (e.g., Executive Functions Performance test - EFPT; ^15,16the Multiple Errands Test - MET; ^17–19; Radial Arm Maze task – RAM, ²⁰) this practice was often revealed time-consuming and not always feasible ²¹.

One of the most widely used performance-based tests is the Naturalistic Action Test (NAT), a valid and reliable test that measures the functional abilities of everyday activities in order to assess any inefficiencies (e.g., mis-reaching an object; ^22–24. The NAT is sensitive to mild functional changes associated with cognitive aging^25,26 and includes tasks of increasing complexity, such as lunch preparation ^22,24. The NAT scoring system evaluates both the steps performed for the task and the errors identified in the action. Recently, Giovannetti and colleagues developed a non-immersive digitalized version of NAT designed with a touchscreen interface in which the scoring was automatically calculated by the software without the need for human coding ²⁷. In a pilot study, conducted by the authors, the results showed a high sensitivity of the test in terms of predicting significant group differences in mild functional difficulties ²⁷. Non-immersive systems include the development of a 2D virtual environment projected on a desktop to reproduce images of the world ⁵. In particular, immersion involves stimulating senses, interactions, and reality's similarity to the stimuli used in virtual environments. This feature can depend on the properties of the technological system used to isolate the user from reality ⁵. Although the characteristics of VR technology can deliver different degrees of immersion in a virtual environment, the sense of presence experienced by participants is the result of a complex interaction between human factors and the proprieties of the VR devices ²⁸. The sense of presence plays an important role in the VR experience, and it has been noted that it is often complicated to find a direct influence on performance in VR tasks ^28,29. The scientific literature makes no definitive claims about the relationship between presence and performance ^30–34. Considering the above, evaluating the sense of presence in the analysis of performances in virtual performance-based tests predicted by cognitive tests becomes important. Whether the sense of presence reported null or weak effects on scores of performance-based tasks, it stands to reason that it could act as a moderator of the cognitive test on the scoring of a specific task delivered in a virtual environment.

In line with the above, the present study aimed to test a virtual immersive version of the NAT (i.e., Virtual Reality Action Test, VRAT), following the same protocol implemented by Chirico and colleagues (2020). Using Head-Mounted Display (HMD) devices, we administered the VRAT to a sample of healthy adults. Participants have been asked to perform the same task (i.e., breakfast) in a cross-over trial in both conditions: virtual vs real. Moreover, we administered a cognitive battery test to assess the participants’ cognitive function. Based on previous research ²⁵, we hypothesized that cognitive tests could predict performance in VRAT. Moreover, we expected that the VR experienced by participants (i.e., sense of presence) could moderate the relationship between cognitive tests and VRAT performance. Finally, in line with the literature ²⁷, we expected significant correlations between performances in both virtual and real tasks and cognitive tests.

2.1 Participants

The study sample was composed of 16 adults. Exclusion criteria for both groups were as follows: non-Italian speaker, current or past neurological disorder or major medical illness (e.g., dementia, traumatic brain injury, schizophrenia, etc.), current psychiatric disorder (e.g., major depression), severe sensory or motor deficit that would preclude interaction with devices. Inclusion/exclusion criteria were evaluated via a self-report questionnaire at the time of recruitment and a brief interview following informed consent obtained from all participants. The study was approved by the Ethical Committee (Department of Psychology of Developmental and Socialization Processes at “Sapienza”, University of Rome). All methods were carried out in accordance with relevant guidelines and regulations and in accordance with the Declaration of Helsinki.

2.2 Procedure

All procedures were identical for all participants in a cross-over design and were completed in a single session. The procedure included the following steps: 1) written informed consent obtained by the participant, 2) training on the VRAT with controllers, 3) testing on the VRAT with controllers and its real version (order counterbalanced), 4) presence questionnaire (only following the VRAT), 5) cognitive tests and other questionnaires.

2.3 Performance-Based Functional Tests

The breakfast task was administered in a highly immersive virtual environment and in a real-world environment, following the administration of NAT²⁴. The breakfast task requires participants to prepare a slice of toast with butter and jelly and a cup of coffee with milk and sugar while seated at a table containing a toaster, two knives, one spoon, butter in a butter dish, sugar in a bowl, a bottle of milk, mug filled with warm water, bread, instant coffee, jelly jar, and a napkin at the central workspace. The shape of the table and the spatial arrangement of objects followed the procedures reported in the NAT manual (NAT Manual; mrri.org). The breakfast task was administered in real and virtual environments. In both conditions, participants were instructed to complete the task, as quickly as possible, and without making errors. They were asked to make their movements as clear as possible and to tell the examiner when the task is finished. Performance in real conditions was video-recorded for scoring.

2.4 Virtual Reality Action Test (VRAT)

The VRAT is an immersive VR task that includes an everyday task (i.e., preparing breakfast) designed to maximize ecological validity by simulating real kitchen and household objects. The VRAT environment includes accurate 3D models, spatial audio, and automatic and real-time collection of movement data. The VR system included the HTC Vive head-mounted display that provided a fully immersive experience in a virtual environment and the controllers that provided tactile feedback through vibration to enable interaction with virtual objects in the VRAT. The VRAT system runs on an MSI Trident Gaming Desktop with 8GB RAM and a GTX 1060 graphic card. The HTC Vive head-mounted display provides users with a fully immersive virtual environment. The system provides visual content through two OLED displays for a total resolution of 2160 × 1200 pixels with a 110- degree FoV and a frequency of 90 Hz. The equipment and software specifications were the same used in a recent case study conducted by Chirico and colleagues¹¹.

During the VRAT, the participants were in a seated position in front of a virtual desk with virtual objects. The controllers were used to interact with the VR environment in which the user’s hand motions are directly mapped to the virtual hand movements. As soon as the virtual hand reaches an object, the object is highlighted to indicate that it is selectable and interactable. Users are instructed to press a trigger button to interact with virtual objects in VRAT and release the trigger to end the interaction. The first phase of the VRAT regarded a VR training session in order to familiarize participants with the virtual environment. The training included four mini-tasks that comprised elements of the breakfast task: (1) toast a slice of bread; (2) spread the jelly on toast; (3) add instant coffee to a cup; (4) add milk to a cup. The examiner controlled the presentation of each mini-task from a monitoring position and could correct errors in object identification or the performance of task steps. The participants were encouraged to ask any questions. After the VR training session, the participants completed a single test trial of the VRAT in which they have been asked to perform the breakfast task. Participants were instructed to complete the test trial in silence, as quickly as possible, without making errors, making their movements as clear as possible. At the end of the task, they were told to stop and declare the test over.

Although the VRAT includes the error monitoring module, performance quality, and accuracy on the real condition and virtual conditions were evaluated by three trained coders, who independently viewed and coded the recordings of the participants’ performances to ensure inter-rater agreement. The performance scores were composed by:

Total errors: incorrect actions (commission), the failure to complete a step (omission), and off-task actions (additions).
Micro-errors: inefficient but not overtly incorrect actions; this category of errors refers to subtle inefficient behaviors in performing the task (e.g., reaching an object not needed for the accomplishment of the step).
Accomplishment score: an accomplishment point was assigned for each task step of the breakfast task completed without error (range = 0–16).

2.7 Virtual Reality measures

After the VRAT conditions, questionnaires about the experience were administered to the participants.

Presence Questionnaire

The participants were asked to complete the Italian version of the Presence Questionnaire ³⁵. The questionnaire, in its original form, comprises 24 items that explore different aspects of the VR experience, rated on a 7-point Likert scale. The factors are distributed as follows: 7 items on Realism; 4 items on the Possibility to act; 3 items on the Quality of the interface; 3 items on the Possibility to examine; 2 items on Self-evaluation of performance. In this study, we used the following factors: Realism, Possibility to act, and Possibility to examine. Reliability was higher than 0.88 for each factor.

Cybersickness Symptoms

The Italian version of the VR Cybersickness Symptoms ³⁶ was proposed to the participants to evaluate two types of side effects caused by exposure to VR: visual effects, such as tired eyes, aching eyes, eyestrain, blurred vision, and difficulties focusing, and physical effects, general discomfort, fatigue, boredom, drowsiness, headache, dizziness, concentration difficulties, and nausea. Participants answered reporting on a 6-point Likert scale the presence of symptoms, higher scores indicate more severe symptoms. The original version of the scale was validated by Ames ³⁷ reporting that the maximum irrelevant difference was set at 0.2.

2.8 General screening, cognitive tests

A trained psychologist administered to the participants the Italian versions of several questionnaires. Questions about general health (e.g., presence of psychiatric, neurological, or oncological conditions; motor or visual problems) were collected to screen the presence of clinical conditions that may interfere with the study. Specific cognitive abilities were assessed as summarized in Table 1 Italian version of questionnaires were administered.

Table 1

Cognitive tests used within the study
Variable	Test	Original Scale citation	Italian Scale used for the study	Validity/Reliability of the instrument
Visual memory	Brief Visual Memory Test Revised (BVMT)	Benedict et al., 1996 ³⁸	Argento et al., 2016 ³⁹	Test-retest reliability r = from 0.60 for Trial 1 to 0.84 for Trial 3
Verbal fluency	Category Fluency	Benton et al., 1984 ⁴⁰	Zarino et al., 2014 ⁴¹	Test-retest reliability r > 0.75
Processing speed	Trail Making Test (TMT) - Part B	Armitage 1946 ⁴²	Gaudino et al., 1995⁴³, Giovagnoli et al., 1996⁴⁴	Not reported
Working memory	Digit Span backward	Weiss et al., 2010 ⁴⁵	Monaco et al., 2013 ⁴⁶	Test reliability = 0.89
Processing speed and Visual perception	Symbol Search (SS)	Weiss et al., 2010 ⁴⁵	Orsini et al., 2013 ⁴⁷	Test reliability = 0.88

2.9 Data Analysis

Descriptive analyses were performed on the collected data. Cognitive test scores were also evaluated by calculating the standardized z-score for the participants considering the normative data. Three trained observers evaluated the real version of VRAT (Real Action Test; RAT) performance (i.e., accomplishment and errors). Where there was a different evaluation between the observers, an agreement was found after discussion according to the NAT manual.

Pearson correlations were performed to compute the bivariate correlations between the key variables of the study, using jamovi software⁴⁸.

In order to test our hypothesis, the models were analyzed by employing Variance-Based Structural Equation Modeling (VB-SEM; known as Partial Least Squares analysis), which was performed with the WARP PLS v.8.0 statistical software⁴⁹. In VB-SEM, measurement error is explicitly modeled through the construction of latent factors, much like a covariance-based SEM analysis. VB-SEM, on the other hand, estimates models using ranked data, which is distribution-free, unlike covariance-based SEM. Model complexity, sample size, and deviations of the variable distributions from normality have less impact on model estimation. According to published criteria for VB-SEM models, VB-SEM analysis can evaluate the model at the measurement and structural levels. At the measurement level, VB-SEM establishes construct validity of the latent factors using the average variance extracted (AVE) and the composite reliability coefficients (ρ), which should exceed 0.50 and 0.70, respectively. AVEs for latent variables support discriminant validity if their square roots exceed the correlation coefficient with other latent variables. At the structural level, VB-SEM estimates the overall adequacy of the set of hypothesized relations among the model constructs using the goodness-of-fit (GoF) index given by the square root of the product of the AVE and average R² for the model with values of 0.10, 0.25, and 0.36 correspond to small, medium, and large effect sizes for model fit, respectively ⁵⁰. The average path coefficient (APC) and average R² (ARS) coefficients provide additional information about the model's adequacy, both of which should be statistically significantly different from zero. Furthermore, the full collinearity variance inflation factor (AFVIF) is used to measure the level of multicollinearity, with values lower than 3.300 indicating that there are no issues with multicollinearity. Missing data were imputed using linear regression interpolation as recommended ⁵¹. According to our hypothesis, we tested two moderation models, evaluating the role of sense of presence as a moderator in the relationship between 1) “Memory Moderation Model”: cognitive tests related to the memory (i.e., BVMT, Category fluency, and Digit span) domain and VRAT performance, and 2) “Processing Speed Moderation Model”: cognitive tests related to processing speed (i.e., TMT, SS) and VRAT performance. We adopted an alpha level of 0.05, however, given the small sample size we also reported a marginal effect considered as < 0.10.

3.1 Characteristics of the sample

On average, participants were 50.1 years old (SD = 2.2; range = 20–80). The sample comprised 54% of females. According to the Jack & Bondi guidelines⁵², only two participants showed a low cognitive impairment (for a complete characterization of participants, see supplemental material Table S1 at the following link: osf.io/rcmyq).

3.2 Cognitive test

Raw cognitive test scores along with age and education-adjusted normative-based z-scores are reported in Table S1 (osf.io/rcmyq). Scores on most tests of specific abilities fell within the average range, including tests of visual memory, verbal fluency, working memory, processing speed, and visual perception.

3.3 Correlations

The correlation matrix (Table 2) showed a strong correlation between VRAT Scores with RAT scores. Specifically, the VR accomplishment shared 55.5% of the variance with Real scores (r = 0.745, p < 0.001), on the same page, VR error and Real error shared 78.5% of the variance (r = 0.89, p < 0.001). Alongside these results, the relation between cognitive tests and both versions of the NAT showed a coherent picture, in which high performances were strongly related to accomplishment in a positive direction and negatively with errors. Specifically, the accomplishment score in VR showed a pattern of strong correlation (r > 0.40) with all of the cognitive tests, while the errors score in VR showed a stronger correlation (r > 0.60) with Symbol Search and TMT, and a lower correlation with memory tests (i.e., Digit span, Category fluency, BVMT). Considering the scores in real, the correlation matrix showed a similar pattern with VR scores, while a lower intensity for the relationship between accomplishment and cognitive tests than VR scores. The micro-errors in RAT correlated positively with errors in both conditions (i.e., VR and Real) and negatively with accomplishments in both conditions. Moreover, the micro-errors in RAT correlated with micro-errors in VRAT showing a Pearson index of 0.41. Concerning micro-errors in VRAT, the correlation matrix showed no significant relationships with the other parameters.

Table 2

Correlation Matrix between key variables of the study
	BVMT	Category Fluency	Digit Span	Symbol Search	TMT	Acc VR	Error VR	Acc Real	Error Real	Micro-Error Real	Micro-Error VR
BVMT	—
Category Fluency	0.392	—
Digit Span	0.258	0.151	—
Symbol Search	0.368	0.492	0.410	—
TMT	0.425	0.450	0.166	0.524	—
Acc VR	0.429	0.560*	0.518*	0.565*	0.466	—
Error VR	-0.380	-0.430	-0.189	-0.624**	-0.698**	-0.685**	—
Acc Real	0.305	0.450	0.297	0.563*	0.142	0.745***	-0.663**	—
Error Real	-0.548*	-0.425	-0.300	-0.713**	-0.619*	-0.581*	0.886***	-0.712**	—
Micro-Error Real	-0.109	-0.112	0.210	-0.333	0.140	-0.217	0.479	-0.446	0.384	—
Micro-Error VR	-0.017	0.126	0.324	0.239	0.142	0.150	0.105	0.180	0.016	0.416	—
Note. BVMT = Brief Visual Memory Test; TMT = Trial Making Test; Acc VR = Accomplishment in Virtual Reality; Error VR = Error in Virtual Reality;

Acc Real = Accomplishment in Real condition; Error Real = Error in Real condition.

* p < .05, ** p < .01, *** p < .001

3.4 Moderation models

According to our hypothesis, the moderation models exhibited acceptable fit: “Memory Moderation Model” (GoF = 0.868; APC = 0.277, p = 0.05; ARS = .783, p = < .001; AVIF = 1.579) and “Processing Speed Moderation Model” (GoF = .786; APC = .422, p = .01; ARS = .649, p < .001; AVIF = 3.483).

The “Memory Moderation Model” showed that the effects of BVMT on the VRAT score were marginally moderated (p = 0.07) by the sense of presence, positively with accomplishments and negatively with errors. All the paths between cognitive test and VRAT performance were significant or marginally significant, and coherent in the direction.

The “Processing Speed Moderation Model” showed that the statistically significant effect of the Symbol search test on VRAT errors was significantly and negatively moderated by the sense of presence. Concerning the direct effects of the cognitive variables on the VRAT performances, the Symbol Search was significantly and positively related with accomplishments, and negatively with errors, while TMT showed a similar pathway, significant only with the errors performance. The path coefficients and the p values of the models are reported in Figs. 1 and 2.

The main purposes of the study were: first, to evaluate the validity of the Virtual immersive version of the NAT; second, to evaluate the effect of cognitive tests in predicting the VRAT performances, considering also the role of sense of presence as moderator.

In the scientific literature, other scholars digitalized the NAT using not immersive technology with good results ²⁷. Following this mainstream, Chirico and colleagues (2020) developed a virtual immersive version of NAT (VRAT) implemented through Head-Mounted display devices, demonstrating the feasibility of the VRAT in a single case study. The results of the study supported the idea of implementing the VRAT even in people with minimal computer experience or with no prior VR exposure, without any significant risk ¹¹. Furthermore, from a descriptive point of view, the results of both conditions virtual and real suggested similar outcomes¹¹. In line with these suggestions, the present study evaluated the concurrent validity of the VRAT and NAT in a cross-over design study, relying on a sample of 16 participants evaluated on memory and processing speed domains. Correlation analyses showed significant relations between VRAT scores (i.e., accomplishments and errors) and 1) RAT scores (i.e., accomplishments and errors), and 2) tests of cognitive abilities. These results suggested the potential validity of the immersive VRAT for function assessment, as the correlations of the scores (i.e., accomplishment and errors) between the different conditions (i.e., VR and real) showed a high Pearson index (> 0.70). This result is in line with previous evidence supporting the validity of the digitalized automated version of the NAT, where the correlation was 0.47 ²⁷. Regarding the micro-errors, the correlation matrix showed different patterns in the two conditions. Specifically, the micro-errors in RAT showed a coherent pattern with the other scores, although the sample size has influenced the p-values of the correlations. Indeed, the micro-errors correlated positively with errors and negatively with accomplishments in both real and virtual conditions. Moreover, the micro-errors in RAT correlated with micro-errors in VRAT. Concerning micro-errors in VRAT, the null relationships reported in the correlation matrix could be attributed to the difficulty of the software detecting micro-errors in a virtual environment. Indeed, the automated scoring was inaccurate in tracking subtle functional difficulties. Automatic scoring was shown to be overly sensitive to the hand movements of the participants, leading to inaccurate results. The progress of technology may improve the accuracy in detecting micro-errors, the authors of the present study are involved in a new protocol in order to fix this issue. Despite the difficulties mentioned above, the software was accurate in tracking errors and accomplishments, showing promising results.

Correlation analyses were performed between RAT scores and cognitive abilities, besides the relation between error and digit span, all variables resulted in statistically significant or marginally significant correlation. Although not all correlations were significant, given the Pearson indices, it is reasonable to think that the small sample size could have influenced these results. In any case, despite the small sample size, these results are in line with the literature ^{24,25,27, 53–55}, highlighting the validity of the RAT test.

As expected, also the correlation matrix between VRAT measures and tests of cognitive abilities followed the same pattern of the RAT’s correlation with cognitive tests, suggesting the accuracy of the automated score in detecting the accomplishments and errors during the task. These results are in line with a previous study ²⁷ related to the validation of a virtual non-immersive version of NAT.

Taken together, these results provided preliminary validity data for the VRAT and suggest the utility of the VRAT as an objective and efficient measure of functional difficulties. Virtual immersive technology allows to develop and implements immersive virtual scenarios in which, unlike non-immersive technology, the users could be able to experience the same feelings as the real scenarios. The role of VR experience is still debated in the scientific literature ²⁸. Specifically, previous studies investigated the role of the sense of presence experienced by participants during the VR tasks ^{32–34, 56}, suggesting that it is often complicated to find a direct effect on scoring ²⁹. For this reason, it becomes important to evaluate the sense of presence experienced by participants testing its role in moderating the effect of cognitive tests on VR tasks, when this can be applied.

To the best of our knowledge, this is the first study that evaluates the role of VR presence in moderating the relationship between cognitive abilities and the score of performance-based tests in virtual immersive environments. Our results showed that the sense of presence experienced by the participants during the task moderated the effects of BVMT on VRAT scores and the relationship between symbol search and errors. Although the relationships between sense of presence and performance were not found in previous studies ^34,56 or were found weak associations ^32,57, our findings highlighted that the VR presence may have a moderation effect in these contexts. On the same page, the highest direct effects in the two models showed a pattern that emphasizes the role of visual components in our VR task. As matter of fact, our results highlight as both cognitive tests, related to the visuospatial domain (i.e., visuospatial memory; BVMT, and visuospatial attention; symbol search) had the highest effects on VRAT performances and were the only to being moderated.

Maneuverer and colleagues (2020) tested the effect of the sense of presence in predicting spatial cognition performance in an immersive virtual test (Rod-and-Frame Test; RFT). The authors found a positive effect of the sense of presence and other variables (e.g., cybersickness, game experience) on the participant's performance. Moreover, the scholars tested a mediation model without any significant results. However, they highlighted the role of human factors (e.g., sense of presence, cybersickness, game experience) in the spatial immersive virtual tasks. Following this claim, our study provides an empirical contribution to the role of sense of presence as a moderator, considering the validation of the VRAT test in performing daily living activities with no spatial performances involved. Our results showed that the sense of presence was not a predictor of VRAT performances. However, correlation analysis showed a null effect in the relation between sense of presence and cognitive domain and VRAT performances. To speculate, these results can be explained by the fact that given the different mechanisms involved in the NAT, where spatial performances are not involved to complete the task, the sense of presence becomes a crucial booster of expressing cognitive abilities (i.e., daily living activities) through VR devices. Moreover, participants reported no symptoms of cybersickness in VRAT suggesting the feasibility of the VR devices for all of them. The present study posed some important questions related to the complex interaction between individual factors and devices (e.g., sense of presence). Our results represented a first step in understanding the process behind the individual variations of sense of presence, without any manipulation of the immersive levels provided by the technology.

Future studies should take into account the interaction between human factors and VR systems, focusing on the relationships between several human factors (e.g., cybersickness, gender, VR experience) and different levels of immersive technology, maybe varying the immersive levels of the virtual environment. Although several studies mentioned above have evaluated the direct effect of human factors on performances in VR tasks, future researchers should consider these factors as a moderator component in the resolution of the task in a virtual environment (e.g., performance-based test in VR). Although the present study showed promising results, given the small sample of participants, broader studies are needed to confirm these findings.

Data availability statement

Data are available at the following link:osf.io/rcmyq

Author contributions

All authors conceived and planned the experiments, contributed to the interpretation of the results, provided critical feedback, and helped to shape the research, analysis, and manuscript. AC, FeG, FrG, PN, and MP carried out the experiments. GDP, PN, and LG programmed the software. AC, TP, MDA, and FA managed and analyzed data. MDA and TP coded visual data. AC and TP took the lead in writing the manuscript. AC, TP, AG, and FL critically revised the manuscript and approved the final submitted version.

Acknowledgment

The research has been funded by the following organizations: Sapienza, University of Rome, Department of Psychology of Social and Developmental Processes; Sbarro Health Research Organization (www.shro.org), PA USA; PNRR, partnership 8, spoke 4 (Trajectories for active and healthy aging). We would like to thank Ms. Angela Romano for coordinating the data collection procedure and recruiting.

Henderson, A., Korner-Bitensky, N. & Levin, M. Virtual Reality in Stroke Rehabilitation: A Systematic Review of its Effectiveness for Upper Limb Motor Recovery. Top Stroke Rehabil 14, 52–61 (2007).
Kuil, M. N. A. van der, Visser-Meily, J. M. A., Evers, A. W. M. & Ham, I. J. M. van der. A Usability Study of a Serious Game in Cognitive Rehabilitation: A Compensatory Navigation Training in Acquired Brain Injury Patients. Front Psychol 9, (2018).
Indovina, P. et al. Virtual Reality as a Distraction Intervention to Relieve Pain and Distress During Medical Procedures. Clin J Pain 34, 858–877 (2018).
Ellis, S. R. What Are Virtual Environments? IEEE Comput Graph Appl 14, 17–22 (1994).
Cipresso, P., Giglioli, I. A. C., Raya, M. A. & Riva, G. The past, present, and future of virtual and augmented reality research: A network and cluster analysis of the literature. Front Psychol (2018) doi:10.3389/fpsyg.2018.02086.
Oliveira, J. et al. Performance on naturalistic virtual reality tasks depends on global cognitive functioning as assessed via traditional neurocognitive tests. Appl Neuropsychol Adult 25, 555–561 (2018).
Coleman, B., Marion, S., Rizzo, A., Turnbull, J. & Nolty, A. Virtual Reality Assessment of Classroom – Related Attention: An Ecologically Relevant Approach to Evaluating the Effectiveness of Working Memory Training. Front Psychol 10, (2019).
Botella, C., Fernández-Álvarez, J., Guillén, V., García-Palacios, A. & Baños, R. Recent Progress in Virtual Reality Exposure Therapy for Phobias: A Systematic Review. Curr Psychiatry Rep 19, 42 (2017).
Plancher, G., Tirard, A., Gyselinck, V., Nicolas, S. & Piolino, P. Using virtual reality to characterize episodic memory profiles in amnestic mild cognitive impairment and Alzheimer’s disease: Influence of active and passive encoding. Neuropsychologia 50, 592–602 (2012).
Chirico, A. et al. Virtual Reality in Health System: Beyond Entertainment. A Mini-Review on the Efficacy of VR During Cancer Treatment. J Cell Physiol 231, 275–287 (2016).
Chirico, A. et al. Virtual Reality for the Assessment of Everyday Cognitive Functions in Older Adults: An Evaluation of the Virtual Reality Action Test and Two Interaction Devices in a 91-Year-Old Woman. Front Psychol 11, (2020).
Nolin, P., Stipanicic, A., Henry, M., Joyal, C. C. & Allain, P. Virtual reality as a screening tool for sports concussion in adolescents. Brain Inj 26, 1564–1573 (2012).
Corti, C. et al. Virtual-Reality Performance-Based Assessment of Cognitive Functions in Adult Patients With Acquired Brain Injury: A Scoping Review. Neuropsychol Rev 32, 352–399 (2022).
Palombi, T., Mandolesi, L., Alivernini, F., Chirico, A. & Lucidi, F. Application of Real and Virtual Radial Arm Maze Task in Human. Brain Sci 12, 468 (2022).
Baum, C. M. et al. Reliability, Validity, and Clinical Utility of the Executive Function Performance Test: A Measure of Executive Function in a Sample of People With Stroke. The American Journal of Occupational Therapy 62, 446–455 (2008).
Katz, N., Tadmor, I., Felzen, B. & Hartman-Maeir, A. The Behavioural Assessment of the Dysexecutive Syndrome (BADS) in schizophrenia and its relation to functional outcomes. Neuropsychol Rehabil 17, 192–205 (2007).
Aldermann, N., Burgess, P. W., Knight, C. & Henman, C. Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society 9, 31–44 (2003).
Burgess, P. W. et al. The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society 12, 194–209 (2006).
Knight, C., Alderman, N. & Burgess, P. W. Development of a simplified version of the multiple errands test for use in hospital settings. Neuropsychol Rehabil 12, 231–255 (2002).
Mandolesi, L., Petrosini, L., Menghini, D., Addona, F. & Vicari, S. Children’ s radial arm maze performance as a function of age and sex. International Journal of Developmental Neuroscience 27, 789–797 (2009).
Chevignard, M. P., Soo, C., Galvin, J., Catroppa, C. & Eren, S. Ecological assessment of cognitive functions in children with acquired brain injury: A systematic review. Brain Inj 26, 1033–1057 (2012).
Giovannetti, T., Libon, D. J., Buxbaum, L. J. & Schwartz, M. F. Naturalistic action impairments in dementia. Neuropsychologia 40, 1220–1232 (2002).
Giovannetti, T. et al. Characterization of Everyday Functioning in Mild Cognitive Impairment: A Direct Assessment Approach. Dement Geriatr Cogn Disord 25, 359–365 (2008).
Schwartz, M. F., Segal, M., Veramonti, T., Ferraro, M. & Buxbaum, L. J. The Naturalistic Action Test: A standardised assessment for everyday action impairment. Neuropsychol Rehabil 12, 311–339 (2002).
Rycroft, S. S., Giovannetti, T., Divers, R. & Hulswit, J. Sensitive performance-based assessment of everyday action in older and younger adults. Aging, Neuropsychology, and Cognition 25, 259–276 (2018).
Seligman, S. C., Giovannetti, T., Sestito, J. & Libon, D. J. A New Approach to the Characterization of Subtle Errors in Everyday Action: Implications for Mild Cognitive Impairment. Clin Neuropsychol 28, 97–115 (2014).
Giovannetti, T. et al. The Virtual Kitchen Challenge: preliminary data from a novel virtual reality test of mild difficulties in everyday functioning. Aging, Neuropsychology, and Cognition 26, 823–841 (2019).
Maneuvrier, A., Decker, L. M., Ceyte, H., Fleury, P. & Renaud, P. Presence Promotes Performance on a Virtual Spatial Cognition Task: Impact of Human Factors on Virtual Reality Assessment. Front Virtual Real 1, (2020).
Nash, E. B., Edwards, G. W., Thompson, J. A. & Barfield, W. A Review of Presence and Performance in Virtual Environments. Int J Hum Comput Interact 12, 1–41 (2000).
Slater, M., Linakis, V., Usoh, M. & Kooper, R. Immersion, presence and performance in virtual environments. in Proceedings of the ACM Symposium on Virtual Reality Software and Technology - VRST ’96 163–172 (ACM Press, 1996). doi:10.1145/3304181.3304216.
Stanney, K. M., Kingdon, K. S., Graeber, D. & Kennedy, R. S. Human Performance in Immersive Virtual Environments: Effects of Exposure Duration, User Control, and Scene Complexity. Hum Perform 15, 339–366 (2002).
Stevens, J. A. & Kincaid, J. P. The Relationship between Presence and Performance in Virtual Simulation Training. Open Journal of Modelling and Simulation 03, 41–48 (2015).
Cooper, N. et al. The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment. PLoS One 13, e0191846 (2018).
Ma, R. & Kaber, D. B. Presence, workload and performance effects of synthetic environment design factors. Int J Hum Comput Stud 64, 541–552 (2006).
Scheuchenpflug, R., Ruspa, C. & Quattrocolo, S. Presence in virtual driving simulators. Human Factors in the Age of Virtual Reality (2003).
Solimini, A. G., Mannocci, A. & di Thiene, D. A pilot application of a questionnaire to evaluate visually induced motion sickness in spectators of tri-dimensional (3D) movies. Ital. J. Public Health.
Ames, S. L., Wolffsohn, J. S. & McBrien, N. A. The Development of a Symptom Questionnaire for Assessing Virtual Reality Viewing Using a Head-Mounted Display. Optometry and Vision Science 82, 168–176 (2005).
Benedict, R. H. B., Schretlen, D., Groninger, L., Dobraski, M. & Shpritz, B. Revision of the Brief Visuospatial Memory Test: Studies of normal performance, reliability, and validity. Psychol Assess 8, 145–153 (1996).
Argento, O. et al. Regression-Based Norms for the Brief Visuospatial Memory Test-Revised in Italian population and application in MS patients. Clin Neuropsychol 30, 1469–1478 (2016).
Benton, A. L. & Sivan, A. B. Problems and conceptual issues in neuropsychological research in aging and dementia. J Clin Neuropsychol 6, 57–63 (1984).
Zarino, B., Crespi, M., Launi, M. & Casarotti, A. A new standardization of semantic verbal fluency test. Neurological Sciences 35, 1405–1411 (2014).
Armitage, S. G. An analysis of certain psychological tests used for the evaluation of brain injury. Psychol Monogr 60, i–48 (1946).
Gaudino, E. A., Geisler, M. W. & Squires, N. K. Construct validity in the trail making test: What makes part B harder? J Clin Exp Neuropsychol 17, 529–535 (1995).
Giovagnoli, A. R. et al. Trail making test: normative values from 287 normal adult controls. The Italian Journal of Neurological Sciences 17, 305–309 (1996).
Weiss, L. G., Saklofske, D. H., Coalson, D. & Raiford, S. E. WAIS-IV clinical use and interpretation: Scientist-practitioner perspectives. (Academic Press, 2010).
Monaco, M., Costa, A., Caltagirone, C. & Carlesimo, G. A. Forward and backward span for verbal and visuo-spatial data: standardization and normative data from an Italian adult population. Neurological Sciences 34, 749–754 (2013).
Orsini, A. & Pezzuti, L. WAIS-IV. Contributo alla taratura italiana (16–69 anni). (2013).
The jamovi project. Preprint at (2022).
Kock, N. WarpPLS User Manual: Version 8.0.. Preprint at (2022).
Tenenhaus, M., Vinzi, V. E., Chatelin, Y. M. & Lauro, C. PLS path modeling. Comput Stat Data Anal (2005) doi:10.1016/j.csda.2004.03.005.
Kock, N. Advanced Mediating Effects Tests, Multi-Group Analyses, and Measurement Model Assessments in PLS-Based SEM. International Journal of e-Collaboration (2014) doi:10.4018/ijec.2014010101.
Bondi, M. W. et al. Neuropsychological Criteria for Mild Cognitive Impairment Improves Diagnostic Precision, Biomarker Associations, and Progression Rates. Journal of Alzheimer’s Disease 42, 275–289 (2014).
Giovannetti, T. et al. Coffee with jelly or unbuttered toast: Commissions and omissions are dissociable aspects of everyday action impairment in Alzheimer’s disease. Neuropsychology 22, 235–245 (2008).
Kessler, R. K., Giovannetti, T. & MacMullen, L. R. Everyday action in schizophrenia: Performance patterns and underlying cognitive mechanisms. Neuropsychology 21, 439–447 (2007).
Nolin, P., Banville, F., Cloutier, J. & Allain, P. Virtual Reality as a New Approach to Assess Cognitive Decline in the Elderly. Academic Journal of Interdisciplinary Studies (2013) doi:10.5901/ajis.2013.v2n8p612.
Pallamin, N. & Bossard, C. Presence, Behavioural Realism and Performances in Driving Simulation. IFAC-PapersOnLine 49, 408–413 (2016).
Youngblut, C. & Huie, O. The relationship between presence and performance in virtual environments: results of a VERTS study. in IEEE Virtual Reality, 2003. Proceedings. 277–278 (IEEE Comput. Soc, 2003). doi:10.1109/VR.2003.1191158.

No competing interests reported.

SupplementarymaterialsVRNAT2.xlsx

Download PDF

Journal Publication

published 17 Aug, 2023

Read the published version in Scientific Reports →

Editorial decision: Major revision
23 Apr, 2023
Reviews received at journal
15 Apr, 2023
Reviewers agreed at journal
31 Mar, 2023
Reviewers agreed at journal
13 Mar, 2023
Reviewers invited by journal
13 Mar, 2023
Editor assigned by journal
03 Mar, 2023
Editor invited by journal
02 Mar, 2023
Submission checks completed at journal
02 Mar, 2023
First submitted to journal
17 Feb, 2023

You are reading this latest preprint version

How individual factors intervene in the expression of cognitive abilities in a virtual immersive task: a validation study of the Virtual Reality Action Test

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Materials And Method

2.1 Participants

2.2 Procedure

2.3 Performance-Based Functional Tests

2.4 Virtual Reality Action Test (VRAT)

2.7 Virtual Reality measures

2.8 General screening, cognitive tests

2.9 Data Analysis

3. Results

3.1 Characteristics of the sample

3.2 Cognitive test

3.3 Correlations

3.4 Moderation models

4. Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1