Signals of memory building-blocks: Pupil-linked brain arousal predicts event segmentation and episodic memory

doi:10.21203/rs.3.rs-4724189/v1

Download PDF

Article

Signals of memory building-blocks: Pupil-linked brain arousal predicts event segmentation and episodic memory

https://doi.org/10.21203/rs.3.rs-4724189/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

We automatically segment the continuous stream of our experiences into events with discrete boundaries, and these events serve then as building blocks for later episodic memories. How our brain creates these discrete event representations remains elusive. Here we investigated the role of noradrenergic (NA) transmission, heavily linked to change-detection, in identifying event boundaries during listening to auditorily presented narratives. Specifically, we investigated how pupil dilation, an indirect physiological proxy of NA activity, tracks event boundaries while listening to short stories of everyday events. In a series of three studies, we could demonstrate that event boundaries trigger significant pupil dilation, and the magnitude of these dilations predicts later memory of narratives. Furthermore, we also showed that pupil dilation magnitude at each time point during the narratives scaled as a function of how frequently a set of independent raters indicated the presence of an event boundary preceding that specific story location. This set of results suggests that NA transmission influences episodic memory encoding by signaling unexpected changes in the flow of continuous experience, which then lead to the identification of event boundaries structuring long-term memory representations.

Biological sciences/Psychology/Human behaviour

Biological sciences/Neuroscience/Learning and memory/Long-term memory

Biological sciences/Neuroscience/Cognitive neuroscience/Attention

episodic memory

event segmentation

pupil dilation

LC/NA system

change detection

Every day of our life provides an overwhelming amount of continuously changing experiences, and one of the most intriguing questions of memory research is what determines the later retention of what we have lived through in the past ^1–5. According to research on event segmentation ^6–9, one important determinant of our memories is event segmentation: the continuous stream of our experiences is automatically segmented into discrete events, and the resulting event-boundaries will structure later retrieval of our experiences. Thus, events serve as building-blocks for the later reconstruction of our specific experiences, an ability termed episodic memory ^5,10,11. Specifically, events determine the temporal organization of later episodic memories ^12–15, and predict later recall of information^13,16–20. Boundaries between subsequent events are initiated by prediction errors, which result from the fact that the model or representation of the environment maintained by our brain is no longer an appropriate description of the world and thus has to be updated^7,8,21. Alternatively, not prediction errors per se, but significant changes in background context drive the identification of event boundaries⁶. The neural underpinnings of event segmentation have also been revealed: although the exact activation pattern varies across studies, in most of the studies, a network including the medial prefrontal cortex, posterior parietal regions, medial visual cortex, the parahippocampal cortex and the hippocampus shows increased activity after event boundaries ^16,22–27. Furthermore, it was also shown that the temporal activity pattern of different cortical regions corresponds to event boundaries^22,28.

Thus, it is suggested that prediction error or contextual changes trigger the detection of a new event ^6–8,21, and a consistent set of brain regions shows increased activation triggered after event boundaries^16,22–27. Importantly, however, the question arises what exact neural mechanism links the detection of prediction errors/contextual changes to the activity of this event segmentation network? What neural network is responsible for detecting significant changes in our experiences and initiating the described neural responses associated with event boundary detection?

One potential candidate for such change detection is the Locus Coeruleus/Noradrenergic (LC/NA) system, which is a network of ascending noradrenergic connections enabling the modulation of different cortical areas by the activity of the LC, a small subcortical nucleus^29–31. By virtue of the widespread cortical connections of the LC, the LC/NA system plays a fundamental role in attentional and decision-making processes^29–33. Importantly, the LC/NA system seems to be crucially involved in the detection of sudden environmental changes: it has been associated with reorientation when task information is no longer adaptive^29,34, with signalizing unexpected changes in the environment³³, and with the resetting of task-relevant neural networks after changing environmental contingencies³². Furthermore, it was suggested that the LC/NA system plays a role in how the internal model of the environment is updated after sudden and significant changes^35–37.

Due to this sensitivity to unexpected changes in the environment, a straightforward hypothesis would be that cortical signals of the LC/NA system are triggering the event segmentation brain network underlying the identification of event boundaries. Notably, LC activity influences or is influenced by several brain areas linked to event segmentation (e.g. hippocampus^31,38–41, posterior parietal regions involved in attentional selection ⁴²). Due to these connections, and its sensitivity to environmental change, activity in the LC might be ideally suited to orchestrate the neural event segmentation response. We set out to investigate this hypothesis by examining how event boundaries affect pupil-linked brain arousal, an indirect proxy of LC/NA system function.

The link between pupil size changes and psychological variables has been established for decades (e.g. emotion^43,44, mental effort^45–47, memory^48,49). These widespread effects are attributed to the fact that pupil size is a real-time readout of different brain networks, and can be viewed as a general index of brain arousal, which reflects activity of different neurotransmitter systems and networks^50–54. Due to the physiological link between the musculature of the pupil and the firing rate of the Locus Coeruleus, pupil size changes have been often treated as an indirect marker of the LC/NA system^29,51,52,55, and aligning with the theories about noradrenergic functions (see above), it was shown to be sensitive to changing environmental regularities: it is related to exploration and attentional reorientation^56–59, to unexpected uncertainty⁶⁰, to surprise and model updating^61–66.

If we assume that pupil-linked brain arousal – at least partly - reflects the activity of the LC/NA system^29,51,52,55, and this system detects sudden environmental changes^32,33,37 and thereby plays a key role in event segmentation, then pupil size changes should be observed after event boundaries and they should also predict later memory of observed events. Some evidence already exists regarding this claim: Antony and colleagues⁶⁷ showed that during the viewing of a basketball-game, participant’s pupils dilated after changes in ball-possession, whereas Smith, Loschky, and Bailey⁶⁸ could show a pupil dilation during event boundaries in short videos depicting simple everyday activities (e.g. planting a flower). Finally, Clewett, Gasser and Davachi¹³ used principal component analysis (PCA) to identify different components of the pupil response, which was recorded following auditory stimulation during picture viewing. The authors found that differences in the magnitude of these PCA components were related to the temporal organization of the pictures’ memories.

Importantly, whereas these findings are promising first steps in justifying the role of pupil-linked brain arousal in event segmentation, the to-be detected event boundaries in the above studies were relatively simple perceptual (location change in auditory stimulation) or conceptual features (ball possession change, videos of simple activities), and these studies did not investigate whether boundary-linked pupil responses are predictive of later memory accuracy.

Thus, the aim of our study was (1) to systematically investigate the role of pupil-linked brain arousal in event segmentation of complex stimulus material and (2) to directly test whether pupil-linked brain arousal during event boundaries predicts later memory accuracy. As pupil size is sensitive to different confounds associated with visual stimulus presentation (i.e. pupil size is affected by changes in luminance and eye-movement, see e.g. ⁶⁹), we opted for auditory presentation and investigated how pupil responses accompany event segmentation during the comprehension of complex auditory stimuli (short stories of everyday events). The role of event segmentation in such narrative comprehension attracted significant attention previously^8,70–72, thus it offers a great opportunity to demonstrate that model updating in high-level, complex cognitive processing is associated with pupil-linked-brain arousal. We predicted that pupil-linked brain arousal is sensitive to event boundaries and also influences later retention of information.

To test this hypothesis, we conducted three experiments (see Table 1), where participants listened to four short stories describing everyday events (e.g. a mother and a child visiting an aquarium or a family preparing for a camping trip, all stories adapted from Bailey, Kurby, Sargent and Zacks⁷³). Each story lasted 6-8 minutes and contained two different types of narrative shifts: in four story locations, there was a change in the identity of the protagonist/character (character shift sentences), whereas in four different story locations, the spatial context of the narrative changed (spatial shift sentences). In Exp 1, (N=36), we instructed participants to detect event boundaries, and so we could identify those text locations, where a high proportion of participants indicated an event boundary. Thereafter, in Exp 2 (N=67), we measured changes in the size of participants’ pupils while they listened to the texts. Unlike Exp 1, no segmentation instruction was given, we only asked our participants to listen to and memorize the contents of the stories (intentional encoding), and they had to answer content-related questions after each story (i.e. a cued recall task). Finally, we conducted a second pupillometric experiment (Exp 3, N=33), which aimed to replicate and extend the results of Exp 2, by using incidental encoding: the experimental instruction lacked any reference to a mnemonic task (incidental encoding), and so memory testing was conducted only after the last story. To broaden the scope of our investigation, we also varied the type of memory retrieval: instead of a cued recall task, a forced-choice recognition memory task was used. We predicted that regardless of the instruction, in Exp 2-3, pupil dilation can be observed after both spatial and character shifts and also after those story locations, where participants identified an event boundary in Exp 1. Furthermore, we also predicted that regardless of the memory task used in Exp 2-3, individual differences in such boundary-linked pupil dilations will predict memory of the stories.

Table 1. Aims and methods. Summary of the three studies’ aims and methodological details

	Exp 1 (N=36)	EXP2 (N=67)	EXP3 (N=33)
Aims	Identification of event boundaries	Demonstration of boundary-linked pupil dilations	Replication with different instruction and memory task
Instruction	Memorize the stories & press SPACE during event boundaries	Memorize the stories	Listen to the stories
Instruction	Memorize the stories & press SPACE during event boundaries	Memorize the stories	Listen to the stories
Memory task	7 questions after each story		Forced-choice recognition after all stories

Experiment 1 - Investigating event segmentation during narrative story listening

To investigate whether the predefined narrative shifts are identified as event boundaries, in Exp 1 (N = 36), we investigated where participants detect event boundaries in the stories. Participants were instructed (1) to memorize spatial locations and characters for later memory test, and (2) to indicate with a single key press if they identified an event boundary. We found that location and character shifts were more often associated with event segmentation behavior (i.e. key press), as compared to those sentences, which Bailey et al (2017) used as control text locations without narrative shifts (see Fig 1a). As can be seen in Fig 1b, key-press percentage increased significantly at both narrative and spatial shift sentences, whereas no such increase was present for the control sentences (see Section 1-2 in the supplementary material for detailed description of statistical analyses).

As a next step, we investigated the temporal distribution of key-presses, irrespective of the narrative shifts. To this aim, we calculated for each second of each story the percentage of participants pressing the segmentation key. Key-press percentage values plotted over time are shown for one of the stories in Fig 1d (blue line, see Fig S2a-c in Section 3 of the supplementary material for similar plots of the other stories). In all four stories, a few time points emerged, where key presses are more frequent (time points with key-press frequency values above 0.15 are marked with green squares in Fig 1d). This pattern of a few outlier data points with high key-press frequency resembles the event-segmentation behavior observed in other studies^73,74, and such locations are usually considered event boundaries. Because of this, locations with key-press frequency values above 0.15 were involved in later pupillometric analyses, as they might indicate the presence of an event boundary. We will refer to these time points as event segmentation hotspots. The mean key-press percentage for these segmentation hotspots was M = 19.45%, SD = 3.01. When key-presses are averaged for each sentence (and not for each second separately), a larger agreement was observed: an average of M = 36.76% (SD = 12.91) of the participants indicated an event boundary during the sentences which consisted an event segmentation hotspot (see grey lines in Fig 1d). We also identified locations in the text, termed no-segmentation spots, where none of the participants pressed the segmentation key (shown with yellow in Fig 1d). The detailed description of how these locations were selected can be found in the method section.

The colored lines at the bottom of each plot in Fig 1c show the location of spatial/character-shifts and control sentences, as defined in the original study by Bailey and colleagues⁷³. As can be seen in Fig 1c-d an Fig S2a-c, there is a moderate overlap between spatial/character narrative shifts and event boundaries detected by the participants (i.e. segmentation hotspots): specifically, we identified 24 segmentation hotspots in the four texts, and 15 of them followed a narrative shift by less than five seconds, indicating that participants might have identified a particular narrative shifts as event boundary. The mean temporal difference between the content-based narrative shift and the subsequent event-segmentation hotspot was M = 1.67 sec (SD = 1.86) for the spatial shifts, M = 2.56 sec (SD = 2.18) for the character shifts, respectively.

To conclude, Exp 1 validated the Hungarian adaptation of the texts and identified several event boundaries where possible boundary-linked pupil dilations can be expected.

Experiment 2 – Investigating boundary-linked pupil dilations

In Exp 2 (N = 67), we again instructed participants to listen carefully to the texts, as later they will have to answer several questions regarding spatial locations and protagonists (this part of the instruction was identical to that given in Experiment 1). In contrast to Exp 1, however, in Exp 2, we did not instruct participant to segment the stories with pressing a key on the keyboard, thus their only task was to listen carefully to the narratives. During story listening, pupil size changes of the participants were recorded.

To examine whether pupil-linked brain arousal is sensitive to event boundaries, we investigated how pupil size is affected by those potential event boundaries and control text locations, which were either taken from Bailey and colleagues⁷³ or were identified in Exp 1. To this aim, pupil size values from one second prior to and 12 seconds after these locations were selected and averaged in 500 msec time bins, for each location type separately (see grand mean values in Fig 2a-b). That is, pupil size values were aligned to these locations of interest, and thus, in the following, we will refer to these locations as alignment points. Thereafter, to represent pupil size change relative to the alignment point, the mean value of the 500 msec before the alignment point was subtracted from each subsequent value.

Locations signalizing spatial or character shifts, as used in the original study of Bailey and colleagues⁷³ and validated for the adapted texts in Exp 1, can be seen as content-driven alignment points, as they reflect the assumptions of the stories’ creators about where event boundaries might lie (note that these assumptions are backed by segmentation behavior observed both in the original study and in Exp 1). In contrast, event segmentation hotspots can be seen as data-driven alignment points, as they were identified based on segmentation behavior observed in Exp 1. If event boundaries are reflected in pupil-linked brain arousal, then we might expect an increase in pupil size after these alignment points. We also identified two sets of control alignment points, where no event boundaries, and thus no pupil dilations was expected. First, as content-driven control alignment points, we used the control sentences from the original paper⁷³, which were defined by the authors as sentences where no narrative shift was present. Second, as data driven control alignment point, no segmentation spots with no key-presses in Exp 1 were used.

As can be seen on Fig 2a-b, pupil size increases after both character/spatial shift sentences and event-segmentation-hotspots, whereas it shows a decreasing trend after control-sentences and no-segmentation spots. These differences were further analyzed using cluster based permutation test, which identifies those time periods (clusters), where pupil size changes were significantly different from zero (see details in method section). We report the beginning and end of these clusters along with the corresponding p-value. Furthermore, significant time periods are also indicated with unicolor lines at the bottom part of the graphs in Fig 2a-b.

Pupil size is significantly larger than the baseline both for the spatial shift sentences (cluster duration: 3-10 sec; p = .002, see red horizontal line on Fig 2a) and also for the event segmentation hotspots (cluster duration: 0.5-6 sec; p = .01, see green horizontal line on Fig 2b). Furthermore, pupil size is significantly smaller than the baseline after no-segmentation spots (cluster duration: 4.5-11 sec; p = .006, see orange horizontal line on Fig 2b).

As a further, more stringent test of whether event boundaries affect pupil size, we compared whether pupil size changes after shift sentences and segmentation-hotspots significantly differ from their respective control-conditions. Again, we used cluster-based permutation testing to identify time periods with a significant difference between the conditions (see methods for details). Significant differences between the conditions are indicated with multicolor horizontal lines on Fig 2a-b. We found that pupil size changes evoked after spatial shifts are significantly larger than pupil size changes after the control sentences (cluster duration: 4-11.5 sec; p = .002, see red-grey horizontal line on Fig 2a). Similarly, pupil size changes are significantly larger for segmentation hotspots, as compared to no segmentation spots (cluster duration: 1.5-10 sec; p = .012, see green-orange horizontal line on Fig 2b). None of the above analyses revealed a significant pupil response for character shifts.

As a next step, we examined whether the significant pupil dilation after spatial shift sentences and event segmentation hotspots predict memory performance. To this aim, we computed mean dilation values for each participant for that time bin where the grand-mean pupil curve had its maximal value (spatial shifts: 6.5-7 sec after baseline; event-segmentation hotspots: 4-4.5 sec after baseline, see details in method). Thereafter, we investigated whether these mean pupil dilation values predict participant’s performance on the tests containing the questions regarding the details of the previously heard story. As can be seen in Fig 2c, no significant correlation emerged in the case of spatial shifts, r (65) = 0.11, p = .38. In contrast, we found a significant correlation between memory performance and the magnitude of the pupil response after event-segmentation hotspots, r (65) = 0.24, p = .04, suggesting that larger pupil dilation after segmentation hotspots predicts better memory of the stories.

To sum up, we could demonstrate boundary-linked pupil dilations in the case of both spatial shifts and event segmentation hotspots, whereas no significant pupil dilation could be observed for character shifts (although a small, nonsignificant increase in pupil size emerged, see Fig 2a). Furthermore, in the case of event segmentation hotspots, the magnitude of the boundary-linked pupil dilation correlated with memory performance.

Experiment 3 – Demonstrating boundary-linked pupil dilations using different task-instruction

In Exp 2, the instruction directed participant’s attention toward the spatial locations and characters, thus pupil dilations following character and location shifts might have been caused not by the presence of an event boundary, but by selective attention toward these classes of narrative stimuli. To exclude this possibility, in Exp 3 (N = 33), we aimed to replicate the boundary-linked pupil dilations observed in Exp 2, this time by excluding any component from the instruction, which would specifically focus their attention to any component of the stories – we merely asked them to listen to the stories, because later they will have to complete tasks associated with them. Furthermore, no questions were presented after the stories, as responding to the questions might have influenced processing of subsequent stories. Instead, after the presentation of all stories, a forced-choice recognition task was conducted with sentences taken from boundary locations.

As can be seen in Fig 3, boundary-linked pupil responses are similar to those observed in Exp 2 – both in magnitude and timing. To test the significance of pupil size changes, the same analyses were run, as in Exp 2. We found significant pupil dilation after both spatial shifts locations (cluster duration: 4.5-8.5 sec; p = .022, see red horizontal line on Fig 3a) and segmentation hotspots (cluster duration: 0-6 sec; p = .014, see green horizontal line on Fig 3b). Similar to Exp 2, pupil size decreased significantly after no segmentation spots (cluster duration: 4-8 sec; p = .032, see orange horizontal line on Fig 3b), and there was a significant difference between pupil size changes triggered by segmentation hotspots and no segmentation spots, respectively (cluster duration: 0-7 .5 sec; p = .004, see green-orange horizontal line on Fig 3b). We also found a nonsignificant tendency for the difference between pupil dilation after spatial shifts and control sentences (cluster duration: 5-9 sec; p = .058, see red-grey horizontal line on Fig 3b). Finally, pupil dilation after character shifts was significantly larger than the baseline (cluster duration: 2.5-9 sec; p = .012, see blue horizontal line on Fig 3b), but it did not differ significantly from pupil size changes seen after control sentences.

Memory performance was tested using a forced choice recognition task. In this task, we presented written version of the shift sentences from the four stories together with a slightly modified version of the same sentence: the task of the participants was to select that version of the sentence, which was presented previously during the story listening task (see details in methods). In average, participants correctly selected the original target sentence in M = 69.70 % (SD = 8.85) of their choices (performance level of 50% reflects random guessing). Similar to Exp 2, we investigated whether the hit rate in the forced choice recognition memory task is predicted by individual differences in the magnitude of pupil dilation during the peak interval (i.e. the time bin with the largest dilation: 8-8.5 sec after baseline for spatial shifts; 3-3.5 sec after baseline for event segmentation hotspots). Replicating the results of the previous study, we again found that pupil dilation after spatial shifts was not related to mnemonic performance (r_s(31) = -.08, p = .64, see Fig 3c), but pupil responses triggered by event-segmentation hotspots was positively correlated with recognition memory performance (r(30) = .39, p = .03, see Fig 3d).

Therefore, we can conclude, that the results of Experiment 2 could be replicated: segmentation hotspots and spatial shifts triggered significant pupil dilations, and individual differences in pupil responses after segmentation hotspots predicted later memory performance of the participants. The modified instruction and memory task ensured that these effects are due to selective attention towards locations/characters triggered by the instruction.

Analysis on aggregate data sets from Exp 1-3 - predicting pupil responses in Exp 2-3 from segmentation behavior in Exp 1

As was evidenced in the previous sections, event segmentation hotspots were associated with the largest pupil dilation, and the magnitude of this dilation even predicted memory performance. Note however, that these event segmentation hotspots were defined as those time points, where in Exp 1 at least 15% of the participants had pressed the segmentation key. This cut-off was determined by visual inspection of the key-press percentage plots (i.e. Fig 1c and Fig S2a-c in the supplementary material). To prove that the demonstrated effects are not due to this arbitrary cut-off, we made an additional analysis, in which we investigated for all time points of all stories, how segmentation behavior (i.e. key-press percentage values) measured in Exp 1 predicted pupil dilation in Exp 2-3.

To this aim, for each story, we averaged the pupil size data of all participant of Exp 2-3 (N=100), to get a mean pupil size value for each second of each story (see Fig 4a). Thereafter, using this aggregated data set, for each second of each story, we calculated the relative change in pupil size for each of the subsequent eight seconds (see Fig 4b). These mean pupil size change values from Exp 2-3 were then paired for each second of each story with the corresponding key-press percentage value from Exp 1 (i.e. what percentage of the participants pressed a button during a specific second of a specific story). Using this data set, we could then investigate how segmentation-behavior measured in Exp 1 for a specific second in a specific story predicts pupil size change in the following eight seconds in Exp 2-3.

As can be seen on the histogram presented in Fig 5a, the distribution of key-press percentage is highly skewed in Exp 1, because its value is zero in half of the cases (i.e. no participants pressed the segmentation key in half of the story time). To enable statistical analysis with this highly skewed variable, we formed six groups, representing seconds with low to moderate to high key-press frequencies (groups are indicated by different colors in Fig 5a, boundaries between groups are indicated under the x-axis in Fig 5a, for details, see the method section).

As a next step, we investigated in the aggregated data set of Exp 2-3, how mean pupil dilation is modulated as a function of the above grouping reflecting different key-press frequency in Exp 1. As can be seen on Fig 5b, large pupil size increase is only present for the last category (red line)– this group contained locations with key-press frequency values above .15 (i.e. it overlapped significantly with the group of event segmentation hotspots). To test the significance of this difference, we investigated how key-press frequency at each second influences pupil size changes 4 second later (see shaded box in Fig 5b) by conducting a nonparametric Kruskall-Wallis test testing the effect of key-press percentage group at baseline, as between-subjects factor, on mean pupil dilation 4 sec later, as dependent variable (note that in this analysis, the unit of analysis is not the participant, but the time point, as we aggregated our data over participants). This analysis revealed that key-press frequency in Exp 1 exerted a small, but significant effect on mean pupil dilation 4 sec later, χ²(5) = 18.99, p = .002, η²= 0.007 (see Fig 5c). Post-hoc Mann-Whitney U-tests with Bonferroni-correction revealed that pupil dilation observed for the largest key-press frequency group (>0.15) was significantly larger than the pupil dilation observed for the first three key-press frequency group (key-press frequency: 0-0.9, all Us > 1195, all zs < -3.3, all ps <.005), whereas no significant difference was found in the case of the fourth and fifth group (key-press frequency: 0.9-0.15, all Us < 654, all zs > -2.41, all ps > .08).

Finally, we also investigated, how key-press percentage, treated as a continuous variable, is associated with pupil dilation 4 seconds later. Note that the majority of the time-points was associated with low key-percentage values (see Fig 5a), thus to control for the potential distorting effect of these data points on the correlation analysis, only data points with key-press frequency values above 0.06 were involved in this analysis. As can be seen on Fig 5d, there was a positive relationship between key-press frequency and pupil dilation 4 second later, r_s (245) = .17, p = .007. The correlation is weaker, but still significant, if all data points are involved, r_s (1802) = .07, p = .003.

To sum up, key-press frequency in Exp 1 predicted the magnitude of the pupil response in Exp 2-3 even when examining all data points of all stories.

In a series of behavioral and pupillometric experiments, we demonstrated that event boundaries in narrative texts evoke significant pupil dilation peaking 3-8 seconds after the event boundary, and some of these pupil responses predict later memory performance for the texts. Specifically, we demonstrated that event segmentation behavior observed in Exp 1 could be used to predict pupil size changes in Exp 2-3. Importantly, we could also show that in Exp 2-3, no pupil dilation was present for those locations, where no participants identified an event boundary in Exp 1. This association was shown also in a second analysis, where we averaged the pupil size data of all participants of Exp 2-3 and showed that the aggregated pupil response for each second scaled as a function of key-press frequency in Exp 1. Finally, we could also demonstrate that the magnitude of pupil dilation after event segmentation hotspots predicted later memory of the texts, as evidenced by using either open ended questions (Exp 2) or a forced-choice recognition task (Exp 3). Regarding the content-based narrative shifts identified previously by Bailey and colleagues⁷³, the results were less straightforward: Spatial shifts, but not character shifts, triggered a reliable pupil response in both Exp 2 and Exp 3, but no correlation with memory performance could be shown, either for pupil responses after character or spatial shifts, respectively.

Our results are in line with our prediction that the identification of event-boundaries is supported by neural circuits underlying pupil size changes. Specifically, pupil responses are linked to the activity of the LC/NA system^29,51,52,55, and this system is suggested to be involved in the detection and handling of changing environmental regularities^29,33,37. Thus, the sensitivity of pupil-linked brain arousal to narrative changes complements previous results showing that pupil-linked brain arousal is associated with the tracking of changing environmental contingencies^56–66. Importantly, most of these previous studies used relatively simple experimental tasks often originating from research on animal behavior. These designs are well-grounded analogues of human behavior and information processing (i.e. reversal learning or four-arm bandit tasks), but our results complement them in an important new way: they show the sensitivity of pupil-linked brain arousal to environmental changes during narrative comprehension, which not only models, but itself is an important facet of human information processing – note that participants listened to the texts in the same way, as they would do this when processing auditory verbatim stimulus outside the lab (e.g. listening to a podcast or to a lecture).

Our results provide further evidence for the important role of NA neurotransmission in episodic memory. It has been shown previously, that the LC/NA system is involved in the contextual reinstatement of episodic memories, and the role of NA neurotransmission in the encoding and consolidation of emotional memories has been also established^75–77. Furthermore, it has been shown that LC activity modulates hippocampal processing^31,38–41, and it has been suggested that the LC-hippocampus link might underlie the effects of novelty and unexpectedness on episodic memory^40,78. Our results might reveal an additional pathway explaining how the LC/NA system might be involved in successful remembering: by contributing to the successful encoding of our experiences by means of efficient event segmentation.

Our results also provide important insights into how noradrenergic transmission might be involved in event segmentation. Previous results already showed that event boundaries might be associated with pupil-linked brain arousal during relatively simple cognitive processing (i.e. watching a basketball game⁶⁷, indoor/outdoor decisions¹³) – our results thus extend these findings by showing that pupil dilation during complex narrative comprehension also signalizes event segmentation. Clewett and colleagues¹³ also showed that differences between different PCA components of the pupil response during the processing of auditory tones serving as event boundaries predicted later temporal organization of episodic memories. We extended this result by showing that the magnitude of pupil responses per se, and not only differences between different PCA components, can be linked to later episodic memory. Furthermore, whereas the results of Clewett and colleagues¹³ support the findings linking event segmentation to temporal organization of episodic memories (i.e. temporal distance, order memory, see e.g. ^12–15, our results show that pupil linked brain arousal might underly the previously demonstrated link between event segmentation and accuracy of later episodic memories^16–20.

Beside the link between pupil-linked brain arousal and event segmentation, our results also raise the possibility that pupillometry can add new insights to the research on narrative comprehension on a more general level. Specifically, we would like to highlight the contrast between the small magnitude of the correlation between key-press frequency and pupil dilation in the aggregated data analysis (Fig 5b-d) on the one hand, and the clear and unequivocal differences between event segmentation hotspots and no-segmentation spots (Fig 2b-3b), on the other hand. This suggest, that whereas event boundaries are clearly related to enhanced pupil responses, so are a variety of other story locations as well. That can be seen, for example, on the upper left part of the scatterplot at Fig 5d, which represents text locations with large pupil dilations and low key-press frequencies. This pattern is off course not surprising knowing that pupil size is a rather general index of neural and cognitive processing^54,69, but it opens up an interesting new field of inquiry: what is the cause for these other dilations? Note that this scatterplot is based on the aggregated data of 100 participants, thus these dilations are unlikely to reflect individual factors (e.g. mind-wandering, bottom-up attention to exogenous distractors in the lab, etc.), and because of this, they might be somehow related to the linguistic and cognitive processing accompanying narrative comprehension at that specific location. An interesting future research area might be to reveal the determinants of these dilations.

Another interesting aspect of our research is the divergence of content based and data driven alignment points. Several, but not all spatial/character shifts were followed by an event segmentation hotspot, and in turn, some, but not all event segmentation hotspots were preceded by a narrative shift (see Fig 2c). Due to low statistical power in our design, we could not investigate relative contribution of narrative switches and other factors signalizing event boundaries (e.g. phrases describing perceptual changes, emotional content or other important aspect of the story line) in pupil dilation. In future research, one might investigate this issue by using stimulus material with more event boundaries, either by presenting stimulus material with more narrative switches and/or using other segmentation instruction (i.e. segmentation based on low-level features, see the distinction between fine-graded and coarse segmentation⁸).

In conclusion, our results reveal the sensitivity of pupil-linked brain arousal to unexpected contextual changes during narrative comprehension, and thus demonstrate the link between pupil size and unexpectedness and surprise in a domain requiring high-level, human-specific cognitive processing. Our findings also suggest a pathway between the LC/NA system and episodic memory: noradrenergic transmission might enhance episodic memory through the detection of unexpected changes in the flow of continuous experience, and these changes will then form the basis of event boundaries structuring long-term memory representations.

Participants

The aim of Exp 1 was to examine in general the pattern of event segmentation behavior in the Hungarian versions of the texts. Because of this, there was no clearly defined effect, which Exp 1 aimed to replicate, and so no formal power analysis was conducted. Because the original study detected the differences between no-shift and shift sentences with a group size of N = 20 (Exp 1, control group), we judged that a sample size of N = 30 would be sufficient for investigating patterns of segmentation behavior. Thus, expecting data loss due to dropouts, the final sample size was somewhat higher, N = 36 (26 women; M_age= 22.86 years, SD_age= 4.36, range: 18 - 40).

In Exp 2, because boundary-related pupil dilations were not reported previously in the literature, we had no prior expectations about the size of this effect. Thus, we used the statistical software G-Power⁷⁹ to compute the sample size required to detect a small to medium effect (Cohen’s d=0.35). The computations were performed for a nonparametric one-sample Wilcoxon signed rank test testing whether the baseline corrected pupil dilation values are significantly greater than zero (i.e. significant dilation relative to the baseline). The required sample size was N = 63 with a power of 0.85 and alpha of 0.05. Thus, expecting dropouts, we collected data from N = 71 participants (45 women; M_age= 21.84 years, SD_age= 1.84, range: 18 - 26).

Finally, the main aim of Exp 3 was to replicate the boundary-linked pupil dilations observed in Ex 2. As a first step for sample size calculation, we used the data of Exp 2 to estimate the magnitude of the effect for pupil responses after spatial shift sentences and event segmentation hotspots, respectively. To this aim, we computed one-sample Wilcoxon signed rank tests testing whether pupil dilation is significantly greater than zero during the peak interval of the two pupil responses (i.e. the time bin with the largest pupil dilation). The magnitude of the effect was medium for both event segmentation hotspots (Cohen’s d = 0.6 for the time bin 4-4.5 sec), and for spatial shift sentences (Cohen’s d = 0.5 for the time bin 6.5-7 sec), respectively. The smaller of the two effect sizes was used to calculate that the required sample size for detecting the boundary-linked pupil dilations is N = 32 (power: 0.85, alpha = 0.05). Thus, expecting dropout, we collected data from N = 39 participants (26 women; M_age= 21.31 years, SD_age= 1.83, range: 19 - 26).

Due to low task performance, missing data or low quality of pupil data, four participants were excluded from Exp 2 and six participant from Exp 3 (see below), thus the final sample size was N = 67 (44 women; M_age= 22.00 years, SD_age= 1.81, range: 18-26) in Exp 2 and N = 33 (24 women; M_age= 21.21 years, SD_age= 1.78, range: 19 – 26) in Exp 3. The experiments were approved by the United Ethical Review Committee for Research in Psychology (Hungary) and were carried out in accordance with the Code of Ethics of the Word Medical Association (Declaration of Helsinki) for experiments involving humans.

Stimuli

We adapted the stimulus set from Bailey and colleagues⁷³. The authors created eight stories of everyday activities (e.g. a mother and a child visiting an aquarium; a family preparing for a camping trip). Each story was designed to contain narrative shifts: in four sentences, there was a change in the identity of the character (character shift sentences). In further four sentences, the spatial location changed (spatial shift sentences). As a control condition, the authors defined four sentences in each story, in which no shift occurred (no shift sentences).

To adapt these stories to Hungarian, first, the eight stories were translated to Hungarian by a professional translator. After the translation, one of the authors (PP) reviewed all stories, and the translated versions were finalized after discussing all open questions regarding terminology and sentence formulation. Minor changes in content were required due to cultural differences (e.g. in one of the original stories, an American football game was mentioned, whereas in the Hungarian story, this was modified to a soccer game). Furthermore, in the original study, the stories were shown on a computer screen. To avoid artefacts in pupil size measurement due to visual processing, we opted for auditory presentation. The stories were read out by a female narrator, and we recorded her voice. These recordings were then played for the participants, who listened it through earphones (Exp 2-3) or the microphone of their computer (Exp 1).

To test whether the narrative shifts are identified as event boundaries also in the Hungarian translations of the texts, we conducted a pilot experiment (N = 32), in which participants listened to all eight stories created by Bailey and colleagues⁷³. Participants were instructed (1) to memorize spatial locations and characters for later memory test, and (2) to indicate with a single key press if they identified an event boundary (i.e. event segmentation behavior). We found that location and character shifts were more often associated with event segmentation behavior, as compared to those sentences, which Bailey and colleagues⁷³ used as control text locations without narrative shifts (see Section 1 in supplementary material for description and results of the pilot study). We selected those four texts, in which the predefined spatial/character shifts most consistently triggered key-presses, and these stories were used in all the three experiments (with respect to their topic, the selected stories are referred to as “aquarium”, “hospital”, “castle”, and “camping”).

To test memory for the stories in Exp 1-2, we created seven questions with respect to each story. These questions focused on specific characters and spatial locations (e.g. “Where did they stop the car”? “What kind of phobia does Julia have?”).

To test memory in Exp 3 with a forced choice recognition task, we created 41 sentence pairs. One of the sentences was always a boundary sentence taken from one of the stories, whereas the other sentence was a slightly modified version of it: always one specific word or phrase was changed, which slightly changed the sentence without significantly affecting the meaning of the sentence (e.g. ‘That day they visited a medieval castle, which was Stephen's idea’. vs. ‘That day they visited a ruined castle, which was Stephen's idea’).

The supplementary material contains the text of all stories (Section 4) and the memory tests as well (Section 5).

Design

In Experiment 1-3, we used the four stories chosen during the piloting work (see above). The order of presentation was counterbalanced: that is, for different participants, the stories were presented in a different order, and each story was presented in each position approximately the same number of times. Thus the mean serial positions of the stories were similar in Exp 1 (aquarium: M = 2.5; hospital: M = 2.5; castle: M = 2.61; camping: M = 2.38), Exp 2 (aquarium: M= 2.52; hospital: M = 2.48; castle: M = 2.49; camping: M = 2.51), and Exp 3 (aquarium: M = 2.49; hospital: M = 2.51; castle: M = 2.54; camping: M = 2.46).

In Exp 1-2, immediately after listening to each story, participants’ memory for the previously heard story was tested. In Exp 3, participants took part in a forced choice recognition task after listening to all four stories. During this task, we presented participants a series of sentence pairs. One of the sentences was always presented previously in a boundary location, whereas the other sentence was a slightly modified version of this sentence. Participants had to select the previously heard sentence.

Procedure

The pilot study and Exp 1 was conducted online. Presentation of the stories/questions and the administration of the responses were programmed using the computer presentation software Psychopy⁸⁰, and hosted online using the Pavlovia software (https://pavlovia.org). Each story and the related memory test were programmed as separate tasks. The research assistant contacted the participants using online chat applications and sent the web-links related to the different stories in the predefined order of the pseudorandomization (see above). At the beginning of each task, the instruction was presented on the computer screen, and it was also explained by the experimenter. Participants were told that they would listen to everyday stories. They were instructed to try to memorize the stories, because they would get questions regarding the characters and locations of the stories. Besides, they were also asked to segment the stories to meaningful events by indicating event boundaries with pressing the SPACE key on the keyboard (see Section 6 of the supplementary material for the full instruction). Participants conducted the tasks on their own laptop or PC.

Exp 2 and Exp3 were conducted in the laboratory. Participants were seated in front of a computer screen and an SMI RED500 remote eye-tracker – we asked them to place their head in a chinrest to restrict head-movements. As a first step, the eye-tracker was calibrated. Thereafter, in Exp 2, we instructed participants to memorize the stories – this part of the instruction was similar to Exp 1. Importantly, however, in contrast to Exp 1, in Exp 2, we did not instruct participant to segment the stories, thus their only task was to listen carefully to the narratives. In Exp 3, we omitted from the instruction the direct reference to mnemonic task, and also all hints towards spatial locations or characters. We told participants only to listen to the texts carefully, as later further tasks will be conducted related to the contents of the task. In both experiments, participants listened to the stories using earphones.

Responses to the questions regarding the stories in Exp 1-2 were presented on the screen of the computer, and participants had to type in their responses. In Exp 3, after listening to the last story, participants were given a spreadsheet, which listed all sentence pairs one below the other. Participants task was to mark the correct version of the sentence, which they listened previously. To make the task easier, the sentences were presented in blocks with respect to the specific story, that is participants knew in which story the correct version of the sentence was presented.

Data processing

Rating of memory performance in Exp 2: Although memory task was conducted also in Exp 1 due to maintaining consistency between design in Exp 1 and 2, memory performance was only analyzed in Exp 2. Here, due to technical problems, all responses of one participant, and responses for one of the stories in the case of another two participants were lost. All other responses were coded by a predefined scoring protocol developed by the first author (P.P.). One point was given for correct responses, and zero point for incorrect ones. For some answers, half a point could be given for partially correct responses. Based on this protocol, all responses were scored by a PhD student, who had no knowledge on design, hypotheses or aims of the study. Twenty-five percent of the responses (data from 21 participants) were also coded by the second author (Sz.Á.). She was unaware of the scores given by the PhD student and followed the same scoring protocol. Between-rater correspondence was high: the two raters gave the same points in 94.55% of the cases (kappa = 0.88). As there were four stories with seven questions per story, the maximum score was 28. Thus, points attained by each participant were divided by 28 to get a percentage score – this memory score was used in all analyses. Mean memory score was M = 0.71 (SD = 0.11) in Exp 2. The performance of two participants was more than 3 SD below the mean of the sample, thus their data were also excluded from further analysis. Thus, in sum, three participants were excluded from further analysis due to missing data or low memory performance (i.e. their pupil data were not analyzed either).

Computing memory performance in Exp 3: We calculated the percentage of sentences, were participants selected the correct, previously heard sentence. For one of the participants, the percentage of correct responses was at chance level (48%), thus his/her behavioral and pupillometric data were excluded from further analysis. After the exclusion, the mean accuracy was M = 0.69 (SD = 0.09).

Segmentation behavior: In the pilot study and in Exp 1, segmentation behavior was assessed by analyzing the frequency and distribution of key presses. For all seconds of all stories, we calculated the percentage of participants who indicated an event boundary. These key-press percentage values were then used to define event boundaries (see next section). Processing of raw key-press frequency data was done using self-written scripts in MATLAB⁸¹.

Alignment points: Because we were interested on how event boundaries affect pupil size, we aimed to select those story locations, which represent event boundaries, and investigate pupil size changes following these story locations. To this aim, pupil size data were aligned to these locations, and thus we will refer to them as alignment points.

First, we identified those sentences that contain a spatial or a character shift, as identified by Bailey and colleagues⁷³. As the sentences were usually multiple seconds long, we selected that time point of the story, when the first information suggesting a character/spatial shift was said by the narrator (i.e. the name of the new character or the first information regarding the new spatial location). Four character- and four spatial-shift locations were identified in each story by Bailey and colleagues⁷³. We refer to these alignment points as content-driven, as they are based on the narrative content of the texts and reflect the assumptions of the stories’ creators about event boundaries.

We also used data-driven alignment points to determine event boundaries – in these cases, we used segmentation behavior of participants in Exp 1 to identify those locations in each story, where key presses indicating event boundaries were most frequent. After inspecting segmentation behavior over time, we selected those seconds where at least 15% of the participants pressed the segmentation key. These locations were termed event segmentation hotspots. Because we aimed to investigate pupil size change in the time window after such changes, we aimed to avoid overlap of pupil data in cases when multiple such segmentation hotspot followed each other by less than ten seconds. Thus, in such cases, only the hotspot with the larger key-press value was included in the analysis. In the case of equal large key-press frequency values within ten seconds, the first location was chosen. No segmentation hotspots were chosen from the first and last ten seconds of the stories. Following the above criteria, we identified 24 such hotspot (aquarium: 6, hospital: 7, castle: 6; camping: 5). The mean key-press percentage for these segmentation hotspots was M = 19.45%, SD = 3.01. Note that this mean frequency value is computed involving key-press frequency for those specific locations (seconds) in the story, where at least 15% of the participants pressed the segmentation key. When all key-presses were counted, which were initiated during the sentences containing an event segmentation hotspot, then larger agreement was observed: an average of M = 36.76% (SD = 12.91) of the participants indicated an event boundary during the sentences which consisted these event segmentation hotspots. Furthermore, the average of M = 48.01% (SD =14.72) of the participant pressed the segmentation key in the period of three seconds before and after the event segmentation hotspots. These agreement values are comparable to those observed for the original stories in the original study⁷³. In the original study, participants could read the written version of the texts and could use a pencil to mark the event boundaries, whereas in our task, they had to identify the boundaries online while listening the stories. Thus, the rather low agreement values at event segmentation hotspots (15-25%) might be due to the fact that the key-presses in our experiment are spread out in time around the actual event boundary – if this is compensated by aggregating responses over multiple seconds, the same level of agreement can be achieved. Note, that pupil-linked brain arousal might be triggered by a discrete time point when a given participant detects change in the incoming input, thus despite the lower mean agreement value of the measure aggregating key-presses on a second-by-second basis, we favored it in the identifying event segmentation hotspots instead of aggregate measures, which sum key-press frequencies over a longer time period.

We also identified two sets of control alignment points, where no event boundaries were expected. First, as content-driven control alignment points, we used the control sentences from the original paper⁷³, which were defined by the authors as sentences where no narrative shift occurred. Here, we chose randomly one word from the control sentence to identify the specific alignment point. Second, to identify data-driven control alignment points, we used data from Exp 1, reflecting segmentation behavior: we identified locations where no key-presses occurred in the current and in the surrounding two (aquarium, castle) or three (hospital, camping) seconds. This difference in criteria was used because of different patterns in the segmentation data: clusters with subsequent zero key-press frequencies were shorter in the former (aquarium, castle), as compared to the latter two stories (hospital,camping). Thus, using the same criterion would have led to widely differing amount of no-segmentation spots in the four stories. When multiple subsequent seconds met this above criteria, then the last second was chosen. When two no-segmentation spots followed each other by less then ten second, then the first one was chosen. We identified 23 such no-segmentation spots (aquarium: 7, hospital: 4, castle: 5; camping: 7).

Importantly, as key-press percentage values were calculated for each second of each story, the alignment points also refer to one specific second of each story (e.g. an event segmentation hotspot was identified in 34th second of the aquarium story). All alignment points are marked in the stories presented in the supplementary material (Section 5).

Pupil size preprocessing: Pupil size data were recorded in mm by the eye-tracker software and were preprocessed for each story and for each participant separately using self-written scripts in MATLAB⁸¹. As a first step, we removed from each data series outliers beyond the physiologically plausible range of 2mm-8mm and also blinks identified by the data processing software of SMI. These data points were interpolated using linear interpolation. As a second step, data were downsampled from the original 500 Hz recording frequency to 10 Hz, then they were smoothed using the Savitzky-Golay filter (parameters: window size=51, polynomial degree=2). Blink frequency (blinks per minute) was M = 17.93 (SD = 14.19) in Exp 2, and M = 19.05 (SD = 15.01) in Exp 3. On average, the percent of interpolated points was M = 7.81 % (SD = 7.11) in Exp 2, and M = 8.52 % (SD = 7.44) in Exp 3 (calculated for the down-sampled data series).

As a second step, we identified data sets with large number of interpolated data (suggesting low data quality). First, we averaged blink frequencies and interpolation ratios of each participant for all four stories. We excluded data from one participant from both experiments, whose average blink frequency was more than 3 SD above the mean of the sample (computed separately for Exp 2 and Exp 3). Furthermore, one participant was excluded from Exp 3, because his/her interpolation ratio was more than 3 SD above the mean of the sample. Then, we checked blink frequencies and interpolation ratios for the individual stories, and data linked to individual stories were also excluded, if these values exceeded 3 SD above the mean of the respective sample. Data from one story was excluded due to excessive blinking in both Exp 2 an Exp 3, whereas due to high interpolation ratio, data were excluded for five stories in Exp 2, and one story in Exp 3. Finally, we also excluded data from those stories, where the mean blink frequency was less than two blinks/minute. Such values are far below the average blinking frequency found in other studies⁸², and might indicate a failure of the blink detection algorithm. Because of this, we excluded data from nine stories in Exp 2, and data of one story in Exp 3. Furthermore, in Exp 3, we also excluded all data from three participants, because their mean blink frequency was below two blinks/min for either three or four stories. Note that when data from a specific story of a participant was excluded due to low data quality, data of the same participants from other stories with appropriate data quality were included in the analysis.

In summary, in Exp 2, data of 67 participants were processed, as four participants were excluded due to low data quality (1), missing data (1) or low memory score (2). These 67 participants completed 268 stories, and overall data from 15 stories were excluded (5.6%). In Exp 3, data of 33 participants were processed, as six participants were excluded due to low data quality (2), low blink frequency (3) or low memory score (1). These 33 participants completed 132 stories, and data from three stories were excluded (2.2 %).

Pupil dilation analysis: Pupil size is sensitive to several psychological and physiological factors and varies considerably with time (see e.g. ⁶⁹). Thus, to investigate how event boundaries affect pupil size, we had to investigate relative change of pupil size compared to specific time points of interest. To this aim, we aligned pupil data to the different alignment points (see above) and investigated how the size of the pupil changes in the subsequent seconds relative to its baseline value immediately before the alignment point. This relative pupil dilation might reflect neural processes underlying information processing associated with the alignment points.

To identify pupil size changes of interest, we extracted data segments starting from 0.5 sec before each segmentation points to 12 seconds after each segmentation point. Then, the average pupil size value was computed for each 500 msec bin of each data segment (i.e. mean pupil size was calculated for the period ranging from 0.5 sec before the alignment point until the start of the alignment point, from the start alignment point until 0.5 sec after the start of the alignment point, from 0.5 after the start of the alignment point until 1 sec after the start of the alignment point, etc.). Thereafter, to represent pupil size change relative to the alignment point, the mean value of the 500 msec before the alignment point was subtracted from each subsequent value. Thus, at the end of this processing, for each participant, and for each alignment point, we had a time series consisting of 25 data points spanning from 500 msec before to 12 secs after the alignment point, which represented pupil size changes relative to the 500 msec before the alignment point.

Statistical analysis

All statistical tests (except cluster based permutation testing) were conducted by self-written R-scripts⁸³. During statistical analysis of the data, we first checked the distribution of the to-be-investigated variables using the Shapiro-Wilk test and depending on the normality of the variables parametric (ANOVA, Pearson- correlation, etc.) or non-normal statistical tests were performed (Wilcoxon Signed Rank Test, Spearman Correlation, etc.). For the correlation analysis in Exp 2-3, we checked and removed outliers whose value exceeded the sample mean by more than 3 SD for the correlation were excluded. Only one outlier had to be removed, from the maximum pupil dilation values linked to segmentation hotspots in Exp 3 (note that even including this outlier results in a significant correlation between pupil dilation variable and memory score, r_s = .37, p = .033).

For comparing the pupil size time series between 0-12 seconds, we used a cluster-based permutation testing, which controls for the Type I error resulting from multiple comparisons of conditions at each of the 24 time bins. We used an MNE-Python script written for this purpose^84,85. During this analysis, first, data are randomly permuted 500 times, and each time a one-sample Wilcoxon test is conducted for each time point, either on the mean of that time bin for a specific condition or on the difference in the means of two to-be-compared conditions (depending on the specific analysis, see results section). Then, significant clusters are identified at each run, that is locations where this statistical test identifies significant differences relative to zero or between conditions (depending on the specific analysis, see results section). Then, z-scores over the different significant clusters are calculated, and a distribution of these summed z-scores over the 500 permutations is created. Only those clusters are then treated as significant, where this weighted score of duration and effect size is above the 95^th percentile of this distribution (corresponding to the 5 % significance level).

Declaration of Competing Interest

The authors declare no conflict of interest.

Funding

This work was supported by the NAP2022-I-2/2022 Hungarian Brain Research Program 3.0 Research Grant and by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the NKFIH K134638 funding scheme. The research reported in this paper is part of project no. BME-NVA-02, implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the TKP2021 funding scheme.

Acknowledgements

The authors would like to thank for Ádám Albi for his help in computing the memory scores of Exp 2.

Data availability statement

The code for the pupil data preprocessing and the code and data for statistical analyses are publicly available on OSF, doi 10.17605/OSF.IO/GPRHS (https://osf.io/gprhs/).

Bartlett, F. C. Remembering: A Study in Experimental and Social Psychology. (Cambridge University Press, Cambridge, England, 1932).
Ebbinghaus, H., Ruger, H. A. & E, C. Memory: A contribution to experimental psychology. (1964).
Markovitsch, H. J. & Brand, M. Forgetting: A historical perspective. in Forgetting (ed. Sala, S. D.) 23–35 (Psychology Press, Hove, 2010).
Schacter, D. L. The Seven Sins of Memory: How the Mind Forgets and Remembers. (HMH, 2002).
Tulving, E. Elements of Episodic Memory. (Oxford University Press, New York, NJ, 1983).
Clewett, D. & Davachi, L. The ebb and flow of experience determines the temporal structure of memory. Current Opinion in Behavioral Sciences 17, 186–193 (2017).
Zacks, J. M. Event Perception and Memory. Annual Review of Psychology 71, 165–191 (2020).
Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S. & Reynolds, J. R. Event perception: A mind-brain perspective. Psychological Bulletin 133, 273–293 (2007).
Wang, Y. C., Adcock, R. A. & Egner, T. Toward an integrative account of internal and external determinants of event segmentation. Psychonomic Bulletin & Review 1–23 (2023).
Conway, M. A. Episodic memories. Neuropsychologia 49, 2305–2313 (2009).
Tulving, E. Episodic Memory: From Mind to Brain. Annu. Rev. Psychol. 53, 1–25 (2002).
Clewett, D., DuBrow, S. & Davachi, L. Transcending time in the brain: How event memories are constructed from experience. Hippocampus 29, 162–183 (2019).
Clewett, D., Gasser, C. & Davachi, L. Pupil-linked arousal signals track the temporal organization of events in memory. Nature Communications 11, 1 (2020).
DuBrow, S. & Davachi, L. Temporal binding within and across events. Neurobiology of Learning and Memory 134, 107–114 (2016).
Rouhani, N., Norman, K. A., Niv, Y. & Bornstein, A. M. Reward prediction errors create event boundaries in memory. Cognition 203, 104269 (2020).
Ezzyat, Y. & Davachi, L. What constitutes an episode in episodic memory? Psychological Science 22, 243–252 (2011).
Flores, S., Bailey, H. R., Eisenberg, M. L. & Zacks, J. M. Event segmentation improves event memory up to one month later. Journal of Experimental Psychology: Learning, Memory, and Cognition 43, 1183–1202 (2017).
Heusser, A. C., Ezzyat, Y., Shiff, I. & Davachi, L. Perceptual boundaries cause mnemonic trade-offs between local boundary processing and across-trial associative binding. Journal of Experimental Psychology. Learning, Memory, and Cognition 44, 1075–1090 (2018).
Sargent, J. Q. et al. Event segmentation ability uniquely predicts event memory. Cognition 129, 241–255 (2013).
Swallow, K. M., Zacks, J. M. & Abrams, R. A. Event boundaries in perception affect memory encoding and updating. (2009).
Nolden, S., Turan, G., Güler, B. & Günseli, E. Prediction Error and Event Segmentation in Episodic Memory. Neuroscience & Biobehavioral Reviews 105533, (2024).
Baldassano, C. et al. Discovering event structure in continuous narrative perception and memory. Neuron 95, 709–721 (2017).
Kurby, C. A. & Zacks, J. M. Preserved neural event segmentation in healthy older adults. Psychology and Aging 33, 232 (2018).
Liu, W., Shi, Y., Cousins, J. N., Kohn, N. & Fernández, G. Hippocampal-medial prefrontal event segmentation and integration contribute to episodic memory formation. Cerebral Cortex 32, 949–969 (2022).
Schubotz, R. I., Korb, F. M., Schiffer, A. M., Stadler, W. & Cramon, D. Y. The fraction of an action is more than a movement: Neural signatures of event segmentation in fMRI. NeuroImage 61, 1195–1205 (2012).
Speer, N. K., Zacks, J. M. & Reynolds, J. R. Human brain activity time-locked to narrative event boundaries. Psychological science 18, 449–455 (2007).
Zacks, J. M. et al. Human brain activity time-locked to perceptual event boundaries. Nature neuroscience 4, 651–655 (2001).
Geerligs, L. et al. A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain. ELife 11, 77430 (2022).
Aston-Jones, G. & Cohen, J. D. An integrative theory of locus coeruleus-norepinephrine function: Adaptive gain and optimal performance. Annual Review of Neuroscience 28, 403–450 (2005).
Mather, M., Clewett, D., Sakaki, M. & Harley, C. W. Norepinephrine ignites local hot spots of neuronal excitation: how arousal amplifies selectivity in perception and memory. The Behavioral and Brain Sciences 39, 200, (2016).
Sara, S. J. The locus coeruleus and noradrenergic modulation of cognition. Nature reviews. Neuroscience 10, 211–223 (2009).
Bouret, S. & Sara, S. J. Network reset: a simplified overarching theory of locus coeruleus e function. Trends Neurosci 28, 574–582 (2005).
Yu, A. Y. & Dayan, P. Uncertainty, neuromodulation, and attention. Neuron 46, 681–692 (2005).
Sara, S. J. & Bouret, S. Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron 76, 130–141 (2012).
Jepma, M. et al. Catecholaminergic regulation of learning rate in a dynamic environment. PLoS computational biology 12, 1005171 (2016).
Muller, T. H., Mars, R. B., Behrens, T. E. & O’Reilly, J. X. Control of entropy in neural models of environmental state. Elife 8, 39404 (2019).
Sales, A. C., Friston, K. J., Jones, M. W., Pickering, A. E. & Moran, R. J. Locus Coeruleus tracking of prediction errors optimises cognitive flexibility: An Active Inference model. PLoS computational biology 15, 1006267 (2019).
Kempadoo, K. A. Dopamine release from the locus coeruleus to the dorsal hippocampus promotes spatial learning and memory. Proc. Natl. Acad. Sci. U. S. A 113, 14835-14840 17 (2016).
O’Dell, T. J. β-adrenergic receptor signaling and modulation of long-term potentiation in the mammalian hippocampus. Learn. Mem 22, 461–471 (2015).
Takeuchi, T. Locus coeruleus and dopaminergic consolidation of everyday memory. Nature 537, 357–362 (2016).
Wagatsuma, A. Locus coeruleus input to hippocampal CA3 drives single-trial learning of a novel context. Proc. Natl. Acad. Sci. U. S. A 115, 310–316 (2018).
Corbetta, M., Patel, G. & Shulman, G. L. The Reorienting System of the Human Brain: From Environment to Theory of Mind. Neuron 58, 306–324 (2008).
Bradley, M. B., Miccoli, L. M., Escrig, M. A. & Lang, P. J. The pupil as a measure of emotional arousal and automatic activation. Psychophysiology 45, 602 (2008).
Oliva, M. & Anikin, A. Pupil dilation reflects the time course of emotion recognition in human vocalizations. Scientific reports 8, 1–10 (2018).
Beatty, J. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin 91, 276–292 (1982).
Kahneman, D. & Beatty, J. Pupil diameter and load on memory. Science 154, 1583–1585 (1966).
Wel, P. & Steenbergen, H. Pupil dilation as an index of effort in cognitive control tasks: A review. Psychonomic bulletin & review 25, 2005–2015 (2018).
Goldinger, S. D. & Papesh, M. H. Pupil dilation reflects the creation and retrieval of memories. Current directions in psychological science 21, 90–95 (2012).
Võ, M. L. H. The coupling of emotion and cognition in the eye: Introducing the pupil old/new effect. Psychophysics 45, 130–140 (2008).
DiNuzzo, M. et al. Brain networks underlying eye’s pupil dynamics. Frontiers in neuroscience 13, 965 (2019).
Joshi, S. & Gold, J. I. Pupil size as a window on neural substrates of cognition. Trends in cognitive sciences 24, 466–480 (2020).
Joshi, S., Li, Y., Kalwani, R. M. & Gold, J. I. Relationships between pupil diameter and neuronal activity in the locus coeruleus, colliculi, and cingulate cortex. Neuron 89, 221–234 (2016).
Reimer, J. et al. Pupil fluctuations track rapid changes in adrenergic and cholinergic activity in cortex. Nat Commun 7, 27824036 (2016).
Strauch, C., Wang, C. A., Einhäuser, W., Stigchel, S. & Naber, M. Pupillometry as an Integrated Readout of Distinct Attentional Networks. (Trends in Neurosciences, 2022).
Murphy, P. R., O’Connell, R. G., O’Sullivan, M., Robertson, I. H. & Balsters, J. H. Pupil diameter covaries with BOLD activity in human locus coeruleus. Human Brain Mapping 35, 4140–4154 (2014).
Jepma, M. et al. Pupil diameter predicts changes in the exploration–exploitation trade-off: Evidence for the adaptive gain theory. Journal of Cognitive Neuroscience 23, 1587–1596 (2011).
Gilzenrat, M. S., Nieuwenhuis, S., Jepma, M. & Cohen, J. D. Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cognitive, Affective, & Behavioral Neuroscience 10, 252–269 (2010).
Pajkossy, P., Szőllősi, Á., Demeter, G. & Racsmány, M. Tonic noradrenergic activity modulates explorative behavior and attentional set shifting: evidence from pupillometry and gaze pattern analysis. Psychophysiology 54, 1839–1854 (2017).
Pajkossy, P., Szőllősi, Á., Demeter, G. & Racsmány, M. Physiological measures of dopaminergic and noradrenergic activity during attentional set shifting and reversal. Frontiers in Psychology 9, 506 (2018).
Pajkossy, P., Gesztesi, G. & Racsmány, M. How uncertain are you? Disentangling expected and unexpected uncertainty in pupil-linked brain arousal during reversal learning. in Cognitive, Affective, & Behavioral Neuroscience 1–22 (2023).
De Berker, A. O. et al. Computations of uncertainty mediate acute stress responses in humans. Nature communications 7, 1–11 (2016).
Lavín, C., San Martín, R. & Rosales Jubal, E. Pupil dilation signals uncertainty and surprise in a learning gambling task. Frontiers in behavioral neuroscience 7, 218 (2014).
Preuschoff, K., M, H. B. & Einhauser, W. Pupil dilation signals surprise: Evidence for noradrenaline’s role in decision making. Front Neurosci 5, 21994487 (2011).
O’Reilly, J. X. et al. Dissociable effects of surprise and model update in parietal and anterior cingulate cortex. Proceedings of the National Academy of Sciences 110, 3660–3669 (2013).
Van Slooten, J. C., Jahfari, S., Knapen, T. & Theeuwes, J. Individual differences in eye blink rate predict both transient and tonic pupil responses during reversal learning. PLoS One 12:e0185665, (2017).
Zénon, A. Eye pupil signals information gain. Proceedings of the Royal Society B 286, 20191593 (2019).
Antony, J. W. et al. Behavioral, physiological, and neural signatures of surprise during naturalistic sports viewing. Neuron 109, 377–390 (2021).
Smith, M. E., Loschky, L. C. & Bailey, H. R. Eye Movements and Event Segmentation: Eye Movements Reveal Age-Related Differences in Event Model Updating. (Psychology and Aging, 2023).
Mathôt, S. Pupillometry: Psychology, Physiology, and Function. Journal of Cognition 1, 16 (2018).
Zwaan, R. A., Langston, M. C. & Graesser, A. C. The construction of situation models in narrative comprehension: An event-indexing model. Psychological science 6, 292–297 (1995).
Zwaan, R. A. & Radvansky, G. A. Situation models in language comprehension and memory. Psychological bulletin 123, 162 (1998).
Zacks, J. M., Speer, N. K. & Reynolds, J. R. Segmentation in reading and film comprehension. Journal of Experimental Psychology: General 138, 307 (2009).
Bailey, H. R., Kurby, C. A., Sargent, J. Q. & Zacks, J. M. Attentional focus affects how events are segmented and updated in narrative reading. Memory & cognition 45, 940–955 (2017).
Zacks, J. M., Speer, N. K., Vettel, J. M. & Jacoby, L. L. Event understanding and memory in healthy aging and dementia of the Alzheimer type. Psychology and aging 21, 466 (2006).
McGaugh, J. L. Make mild moments memorable: add a little arousal. Trends Cogn. Sci 10, 345–347 (2006).
Cahill, L. Beta-adrenergic activation and memory for emotional events. Nature 371, 702-704 135 (1994).
Strange, B. A. & Dolan, R. J. β-Adrenergic modulation of emotional memory-evoked human amygdala and hippocampal responses. Proc. Natl. Acad. Sci. U. S. A 101, 11454–11458 (2004).
Rouhani, N., Niv, Y., Frank, M. J. & Schwabe, L. Multiple routes to enhanced memory for emotionally relevant events. in Trends in Cognitive Sciences (2023).
Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods 39, 175–191 (2007).
Peirce, J. W. et al. PsychoPy2: experiments in behavior made easy. Behavior Research Methods 10 3758 13428-018-01193- (2019).
The MathWorks, Inc. MATLAB. The MathWorks, Inc. (2023).
Jongkees, B. J. & Colzato, L. S. Spontaneous eye blink rate as predictor of dopamine-related cognitive function – A review. Neurosci. Biobehav. Rev 71, 58–82 (2016).
R Core Team. R: A Language and Environment for Statistical Computing. (R Foundation for Statistical Computing, Vienna, Austria, 2024).
Gramfort, A. MEG and EEG data analysis with MNE-Python. Front. Neurosci. 7, (2013).
Larson, E. et al. MNE-Python. Zenodo https://doi.org/10.5281/ZENODO.8322569 (2023).

There is NO Competing Interest.

SupplementaryMaterialFINALSUBMITTED.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Signals of memory building-blocks: Pupil-linked brain arousal predicts event segmentation and episodic memory

Status:

Version 1

Abstract

Figures

Introduction

Results

Discussion

Methods

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1