Prior to emotional decoding, we analyzed the behavioral data to verify the alertness of participants during the fMRI experiment (See Methods). Utilizing data from the simple audiovisual task, we calculated mean accuracy as detecting target signals (hearing a beep or seeing a change in color) within a 2-s window. Data revealed a mean accuracy of 88.2 ± 2.9% across 288 trials (4 trials/movie x 24 movies x 3 times repetition), indicating that participants were sufficiently engaged and attentive throughout the session.
We then analyzed the subjective ratio of emotional driven stimulus that was taken after the fMRI experiment. Interestingly, one-way ANOVA showed that the distribution of these subjective ratings in the behavioral experiment outside the fMRI scanner differed significantly from the pre-experiment movie assignment (F [23,264], 1.57; p, 0.04; partial η2, 0.11) (Fig. 2a, top bar graph). Therefore, we divided the movies into two categories, AD and VD (Fig. 2a, bottom bar graph), which served as labels for training the fMRI decoding models (Fig. 1).
fMRI decoding analyses showed significantly higher mean decoding accuracies in the auditory area (trained and chance-level: t(11), 4.73; p, 0.00031; r, 0.82) and insula trained and chance-level: t(11), 3.55; p, 0.0023; r, 0.73), as compared to chance-level. Conversely, the mean accuracies in the visual areas and amygdala were not significant (Fig. 3).
To rule out the possibility of chance, we compared the mean accuracies of the trained model with that of random model, which was made using random labels. As compared to the random model, significantly higher mean accuracies in the auditory area (trained and random: t(11), 2.81; p, 0.0085; r, 0.65) and insula (trained and random: t(11), 2.36; p, 0.020; r, 0.58) were observed in the trained model. For further confirmation in the off-chance that using only the mean subjective evaluations influenced the results (Fig. 2a, bottom bar graph), we also calculated the mean accuracies for both brain regions using individual subjective evaluations, revealing similar tendencies as with mean subjective evaluations (See Supplementary Information). Given these findings, brain activities in the auditory area and insula appeared to represent which stimuli (auditory or visual) evoked emotional responses during film viewing.