Relating the past with the present: 
Information integration and segregation during ongoing narrative processing

doi:10.21203/rs.3.rs-65710/v1

Download PDF

Article

Relating the past with the present: Information integration and segregation during ongoing narrative processing

https://doi.org/10.21203/rs.3.rs-65710/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 01 May, 2021

Read the published version in Journal of Cognitive Neuroscience →

Version 1

posted

You are reading this latest preprint version

This study examined how the brain dynamically updates event representations by integrating new information over multiple minutes while segregating irrelevant input. A professional writer custom-designed a narrative with two independent storylines, interleaving across minute-long segments (ABAB). In the last (C) part, characters from the two storylines meet and their shared history is revealed. Part C is designed to induce the spontaneous recall of past events, upon the recurrence of narrative motifs from A/B, and to shed new light on them. Our fMRI results showed storyline-specific neural patterns, which were reinstated (i.e. became more active) during storyline transition. This effect increased along the processing timescale hierarchy, peaking in the default mode network. Similarly, the neural reinstatement of motifs was found during part C. Furthermore, participants showing stronger motif reinstatement performed better in integrating A/B and C events, demonstrating the role of memory reactivation in information integration over intervening irrelevant events.

Cognitive Neuroscience

narrative processing

storylines

narrative motifs

information integration

Real-life events unfold over multiple minutes. Using real-life stimuli such as stories and movies, previous studies have revealed a cortical hierarchy of timescales that synthesize information over increasing temporal receptive windows (TRWs) ^1–5. We thus proposed a process-memory model (Hasson, Chen, & Honey, TiCS, 2015)⁶. Unlike classic theories of working memory, which distinguish between areas that process the incoming information and working memory buffers that accumulate and protect the processed information^7,8, in our model, all cortical areas actively sustain memories while dynamically synthesizing them with newly arrived input at their preferred timescales. Namely, early sensory areas integrate information over short timescales of tens of milliseconds, coinciding with the duration of phonemes and words. Adjacent areas along the superior temporal cortex integrate information over hundreds of milliseconds, coinciding with the duration of single sentences, while high order areas, which overlap with the default mode network (DMN) ^9,10, integrate information across paragraphs as the narrative unfolds over many minutes. This framework illustrates a simple recurrent mechanism for continuous event updating in long processing timescale areas at the top of the hierarchy. However, in real life, we often have to integrate discontinuous pieces of information to develop a full understanding of an event. This raises the question of how areas with long processing timescales integrate incoming information with relevant past events, while, at the same time, preserving and protecting the accumulated information from being integrated with irrelevant current events.

To probe this question, we collaborated with a professional author (C.L) to craft an original fictional story with a purposefully designed narrative structure. The first part of the narrative consisted of two seemingly unrelated storylines, A, which takes place in Los Angeles, employing a distinct set of characters, and B, which takes place in New York and involves another set of characters (Fig. 1). The two storylines were presented in an interleaved fashion over 30 segments, 15 segments for each storyline (A₁B₁A₂B_2…A₁₅B₁₅). In the last 15 segments (Part C), characters from the two storylines meet in New York and their shared history is revealed. In other words, C part updates the two storylines with new information previously unknown to the audience. One of the main techniques for bridging part C with A and B was to embed specific images/situations/phrases, i.e., narrative motifs, within either the A or B storylines. The recurrence of these motifs in part C is designed to reinstate specific moments from storylines A and B. For example, in segment A₁, the main character, Clara, makes homemade chili for her husband in LA. In part C, Clara eats and comments on a B storyline character’s (Steven’s) homemade chili recipe; this design attempts to reactivate the memory of segment A_1, making the listeners realize that she has known Steven before moving to LA, and augmenting their understanding of a prior relationship between these two characters. To develop a full understanding of this story, the listeners need to piece together a series of clues (motifs and storyline) like a detective.

We hypothesize that, during the interleaving A and B parts of the story, the accumulated information about storyline A is segregated from incoming information about storyline B via distinctive, storyline-specific neural patterns. Moreover, during the switch from one storyline to the other, we hypothesize that the neural pattern of the relevant storyline is reinstated (becomes more active), while the neural pattern of the other storyline subsides (becomes less active). Similarly, we hypothesize that the recurrence of motifs from the two storylines in part C would induce spontaneous recall of related past events.

Furthermore, we seek neural evidence for the integration of the reinstated events and the new input. We hypothesize that successful reinstatement will help the subjects to better integrate A/B and C events related to the same motif. To test this, we correlated the degree of motif reinstatement with a behavioral index of event integration (assessed with a post-scan test) across subjects. In addition, we predicted that the within-storyline integration would lead to increased difference between the two storylines as the story unfolds. In other words, each new segment would update the storyline it belongs to and push the two storylines further apart from each other, resulting in a storyline x time interaction effect.

Finally, we also examined whether hippocampus plays a role in reactivating recent events, by testing whether higher hippocampal-cortical inter-subject functional correlation (ISFC)¹¹ is associated with stronger storyline reinstatement at storyline transition. In a previous study³, we observed stimulus-driven ISFC between hippocampus and DMN when the participants resumed a movie after a 1-day break. Here, we tested whether hippocampal-cortical connectivity will also facilitate the reinstatement of recent memories from minutes ago as listeners switch from one storyline to the other.

Several studies have used stimuli partially similar to our stimuli to address different research questions. Lahnakoski et al. (2017)¹² also interleaved two independent movies to create their stimulus. This study shows that breaking the flow of a narrative (by interleaving) disrupts the accumulation of information relative to continuous viewing, similar to our previous studies^1–3. However, this study did not test for the coding of storyline-specific information and its reactivation across interruptions. Milivojevic et al. (2016)¹³ used an audio-visual movie with interleaved storylines (“Sliding Doors”) and looked for coding of storyline information. A key limitation of the design of Milivojevic et al. (2016) ¹³ is that storyline differences were confounded with sensory differences (e.g., related to some locations being more prevalent in one storyline than the other). Milivojevic et al. (2016) sought to control for these differences by regressing out sensory features, but this strategy is not ideal: When storyline and sensory information are strongly correlated, regressing out sensory information can attenuate legitimate storyline effects. Our study resolves this confound because both storylines were presented auditorily by the same speaker and thus do not differ in sensory properties. Kauttonen et al. (2018) adopted the movie “Memento”, in which recurring cues were embedded to trigger memory recall, similar to the motifs in our story. However, in the movie “Memento”, the exact same scene is repeated. Therefore, pattern similarity between matching cues could be largely due to the same sensory inputs, as suggested in Kauttonen et al. (2018). In contrast, in our study, recurring motifs are embedded in different scenes, which could better reveal the neural reinstatement of past events that are relevant to, but not identical to, the current scene, thus complementing and adding new information to the literature.

To summarize, this study is optimized to address a critical mechanism of process memory: how incoming information is being segregated from irrelevant recent memories while at the same time, being integrated with relevant recent memories, as the story unfolds over time. By collaborating with a professional writer, we had tight control over the structure of the story. This allows us to shape the moments of storyline transition and the reinstatement of past events in the C part of the story. This unique design allowed us to examine how related events can be preserved and actively integrated in the presence of minute-long intervening irrelevant inputs.

fMRI data were collected from 25 subjects while they listened to a structured narrative that lasted for approximately 1 hour. The narrative has two interleaved, seemingly unrelated storylines, A & B, that converge in the later C part. In the first set of analyses, we tested how ongoing information from each of the two unrelated storylines was accumulated across minute-long segments while being segregated from the parallel unrelated interleaved storyline. In the second set of analyses, we tested how events in part A & B were reactivated in part C. The two storylines are connected to part C using 28 specifically designed, recurrent narrative motifs. These motifs are planted at specific, strategic moments of the narrative by the author (58 occurrences in parts A & B, and 36 occurrences in part C). Participants’ overall comprehension and their understanding of the relations created by these motifs were assessed based on post-scan questionnaires. Using representational similarity analysis (RSA) ^14,15 on brain activation patterns within ROIs independently defined by a whole-brain parcellation of resting-state fMRI¹⁶, we tested whether the structure of the story induced the reinstatement of storylines and narrative motifs, and how this related to the integration of the storylines (assessed behaviorally).

Overall comprehension of the story

The comprehension scores were evaluated based on 28 multiple choice questions in a post-scan questionnaire. The group overall comprehension score was 91% (with a range of 64%-100% across listeners), indicating that most subjects were engaged with the story and were able to follow the plot well.

Relation score of narrative motifs

To test whether participants showing stronger neural reinstatement also integrated related events better, we evaluated the relation between part A/B and C created by motifs using 15 open questions in the post-scan questionnaire. The participants’ answers were rated by the author (Lazaridi) (score 1= voluntary report of both C and A/B events; score 0 =only A/B or C event was reported; none was reported; the wrong relation was reported).

Below are two sample questions and real answers from the participants (see Supplementary Table 3 for all the answers and scores):

The following prompts are words, sentences, or phrases that recurred in the story. Please explain their significance to the story:

Home made chili=?

Ans. (score 1): Steven's tradition that Clara adopts.

Ans. (score 0): Steven makes it.

Ans. (score 0): Clara eats Gary's homemade chili.

Mustard stained blouse =

Ans. (score 1): Margaret stains her blouse with mustard when meeting with Alexander, also Steven can't bear to wash it after Margaret dies.

Ans. (score 0): Margaret's blouse is stained when she meets with Alexander for the first time

Ans. (score 0): Margaret's blouse that Steven can't bring himself to clean.

Neural reinstatement of storyline

We first examined whether, and if so, where in the brain the two seemingly unrelated storylines (A&B) had distinct cortical representations. Using RSA, we compared the neural patterns within each storyline (AA & BB) to the neural patterns between the two storylines (AB). Within each ROI, we averaged over time within each segment (lasting approximately one minute) to extract a spatial pattern of activity for that segment. We then compared pattern similarity between segments from the same storyline to pattern similarity between segments from different storylines (Fig. 2a). Note that all of these comparisons were done between participants (see methods).

Higher within-storyline pattern similarity was revealed in a large set of regions, including language areas (superior/middle temporal gyrus, inferior frontal gyrus, and supplementary motor cortex), areas in the default mode network (including PCC, precuneus, mPFC, SFG, posterior parietal cortex, angular gyrus, posterior hippocampus, and parahippocampal cortex), areas in the executive network, (including anterior insula, MFG, MCC, and supramarginal gyrus), high order visual areas (including cuneus and fusiform gyrus), and subcortical areas (including putamen, thalamus, and caudate; please see Supplementary Table 5 for all the brain area abbreviations we used). We also observed the separation between storylines in anatomically defined hippocampus ROIs (Supplementary Fig. 4)¹³.

We demonstrate the time course of storyline effect at the segment boundary in the region where the largest separation across the two storylines was found, i.e. PCC/precuneus. We computed the pattern similarity between each of the -40~40 TRs around segment boundaries and the typical A or B storyline patterns. As shown in Fig. 2b, at the boundary between B and A segments, the similarity to the typical B pattern rapidly dropped, while the similarity to the typical A pattern increased. The two waveforms crossed around the boundary. Similar results were obtained for the complementary transition from A to B segments (Supplementary Fig. 3).

It is worth noticing that, although the two curves in Fig. 2b seem to be symmetrical with respect to zero, that does not mean that the two storylines had opposing activation patterns. The two patterns are forced to average to approximately zero by the need to subtract the global mean response before computing the typical A/B patterns^19,20. Therefore, the correlation values only reflect the relative, but not the absolute, difference between storylines.

Stronger neural reinstatement of storyline in areas with longer processing timescales

We used an independent dataset ¹ to generate a temporal receptive window (TRW) index for each ROI, i.e. the difference in inter-subject correlation between an intact story and its scrambled version. Higher TRW indices were found in prior studies to be associated with increased capacity to accumulate information over long-timescales ^1,5. If the storyline effect only reflected a difference in low-level properties such as wording or acoustic features (note that the same narrator read all segments), regions with low TRW, i.e. regions insensitive to word scrambling, should also show a storyline effect as strong as in high TRW regions. On the contrary, we found a significant positive correlation between TRW index and storyline effect (Fig. 2c). In other words, areas that are capable of accumulating information over long-timescale had a larger difference between storylines.

Storyline x Time effect

We predicted that the segregation of the two storylines (A & B) should increase as the story unfolds and subjects accumulate further information about the unique context of each storyline. To test this hypothesis, we examined whether the storyline effect increased over time by dividing the AB part into the early and later halves (Fig. 3). Within areas showing the separation between storylines, an increase in the separation of patterns at the later phase (leading to a significant interaction between time and storyline) was found in PCC/precuneus, left AG/IPL, left SFG, right IFG, right MTG/MOG, and right SPL.

We compared the storyline effect in the early and late time bins to avoid imposing assumptions on the time effect (e.g. linearity). Having said this, a graded time x storyline effect is observed in similar regions (Supplementary Fig. 5).

Neural reinstatement of narrative motifs

Information in the C part sheds new light on both A and B events, e.g. Clara learned the chili recipe from Steven; Margaret’s mustard-stained blouse now reminds Steven of her death. We examined how past information from part A/B was reinstated during part C upon the recurrence of the narrative motifs. For each occurrence of motifs in the story, we averaged the 5 TRs after its onset. Then, we correlated each reoccurrence of a narrative motif in part C with all its occurrences in part A or B (Fig. 4). The correlation between matching motifs was computed, as well as the correlation between non-matching motifs from the same storyline (shared storyline) and the correlation between non-matching motifs from the competing storyline (unrelated segments).

Compared to non-matching motifs from the same storyline, the reappearance of the narrative motifs in part C reinstated specific neural patterns seen when the motifs were encountered during the A/B segments in PCC/precuneus, bilateral clusters in posterior temporal lobe/inferior parietal lobes/higher visual areas bilateral lateral frontal areas, and dmPFC (Fig. 5).

Furthermore, TR by TR analysis around the onsets of narrative motifs in part C showed that the correlation rapidly increased after motif onset and lasted for 4-7 TRs, approximately for 3-6 sentences (Fig. 5, upper and lower panels). The reinstatement effect was specific to matching motifs and was not seen between non-matching motifs, either within or across storylines.

Narrative motifs vs. high-frequency word effects

To make sure that the reinstatement of patterns after motif onsets reflects the retrieval of narrative information (as opposed to simple reactivation of word representations shared between the A/B and C segments, e.g., the representation of the word “chili”), we performed the same analysis on a set of high-frequency words that occurred in the C part and in either the A or B storyline (e.g., “watch”). We analyzed 28 high-frequency words to match the number of narrative motifs. If the neural reinstatement effect that we observed for motifs simply reflected the reactivation of word representations, the same effect should be observed when we look at the repetition of high-frequency words like “watch” that have no particular narrative significance. In all ROIs showing a significant motif effect, beside the dorsal PCC, the correlation between matching items was significantly higher for the narrative motifs compared to the high-frequency words, which hovers around zero (p < .05, FWE corrected, Fig. 6). This indicates that word repetition alone was not sufficient to drive the motif reinstatement effect we observed; rather, the words had to refer to significant narrative events (as is true for “chili” but not for “watch”).

Correlation between motif reinstatement and the behavioral relation score

We next asked whether the reinstatement of motif-specific activation patterns led to the integration of A/B and C events. The neural reinstatement triggered by motifs in part C was correlated with the subjects’ ability to relate part C with storyline A and B, as revealed in the relation test (see methods section). In short, each subject was asked to freely recall events related to a given motif from different parts of the story. Within ROIs showing a significant motif effect, including left IPL, left supramarginal gyrus, left angular gyrus, dmPFC, (one-tailed, FDR corrected q < .05, Fig. 7), we observed a correlation between the neural reinstatement of motifs and the individual relation scores. In other words, participants who showed a stronger neural reinstatement of motifs were also better at reporting the narrative related connections among separate events sharing the same motifs.

Narrative motifs, emotional engagement, and memorability

Lazaridi, the author of the story, embedded the motifs in highly emotional scenes and provided emotional weightings for the occurrences of motifs using a 5 point scale, predicting that heightened emotional context would contribute to the memorability of a motif and the motif relation effect. We computed the Pearson correlation between those emotional weightings of motif occurrences in the A/B part and the behavioral relation scores of the fifteen narrative motifs, which have corresponding questions in the behavioral questionnaire. A non-significant tendency toward a positive correlation was found (R = .42, p = .06) (Supplementary Fig. 11).

In addition, to verify that the motif effect does not only reflect matching emotional weightings, we ran the motif RSA after regressing out the effect of emotional weighting match (the product of two emotional weightings) on the pattern similarity between motifs. The results (Supplementary Fig. 12) are very similar to Fig. 5, indicating that the motif effect does not only reflect emotional saliency match.

Hippocampal-cortical ISFC and cortical reinstatement of storyline and motif

To examine whether storyline reinstatement is dependent on connectivity with the hippocampus, we examined the correlation between storyline effect and hippocampal-cortical inter-subject functional correlation (ISFC)¹¹ in ROIs showing a significant storyline effect. For each A/B segment (except for the first segment of each storyline), we correlated the storyline effect with hippocampal-cortical ISFC during that segment and ran the correlation across segments (within participants) and participants (within segments). Because we did not have strong predictions about the relevant time windows for computing the storyline effect and ISFC, we ran an exploratory grid search across a range of analysis parameters. The ISFC between hippocampus and mPFC showed a strong correlation with the storyline effect (FDR corrected q < .05 across ROIs) for multiple settings of analysis parameters (Supplementary Fig. 13), although the result did not survive multiple-comparisons correction when factoring in the full set of analysis parameters, so it should be interpreted with caution. We also examined the correlation between hippocampal ISFC and motif reinstatement in ROIs showing a significant motif effect but did not find a significant correlation.

For this study, we actively designed a structured narrative in collaboration with a professional author to test how related events are dynamically and flexibly integrated by the brain while being protected and segregated from intervening irrelevant events. Our results indicate that the memory traces of recent events can be reactivated as a function of current input. This is seen in Fig. 2b, where the neural patterns associated with the current storyline were reactivated at segment boundaries, while the activation patterns of the irrelevant storylines subsided. This effect is stronger in areas with longer processing timescales, peaking in the DMN (Figure 2n and 2c). The reinstatement of relevant past events was also tested in part C by using motifs to reactivate and update particular moments from both storylines (Fig. 5). As predicted, the presentation of specific motifs in part C triggered the reinstatement of associated A/B events. Taken together, these results revealed a dynamic shift between currently active context and latent inactive contexts, which helps to integrate information over minute-long interruptions while protecting the accumulated information from irrelevant input.

Each storyline/scene is a unique combination of multiple narrative elements, such as characters, locations, goals, etc. In a prior paper (Yeshurun et al, PNAS, 2017) ⁵, we showed that local differences in narrative elements (e.g. switching 2-3 words in a sentence with their antonymous) are amplified in the DMN, and are robust to spatial and temporal blurring. Furthermore, our prior work has shown that at least 15 dimensions of information are encoded in DMN activation patterns shared across subjects (Sherlock, Chen et al., 2017) ¹⁷. Therefore, we speculate that the difference between the neural representations of the two storylines was driven by the unique combinations of narrative features, and should not be attributed to any single narrative dimension in isolation.

Using an audiovisual movie with interleaving storylines, a recent study (Milivojevic et al., 2016)¹³ reported the emergence of storyline-specific patterns in the hippocampus. However, as noted earlier, storyline information in that study was confounded with sensory features, complicating the analysis; our study avoided this confound by presenting both storylines auditorily by the same speaker. Our study not only replicated the finding of neural differentiation in the hippocampus (Supplementary Fig. 4) but also revealed similar patterns of results in extended cortical areas, including language areas, the default mode network, the executive network, high order visual areas, and subcortical areas.

In line with the idea that the reactivated information is integrated with new input, we found that participants showing stronger neural reinstatement of motifs in left SMG, AG, IPL, and dmPFC also gained higher relation scores (Fig. 7). This finding supports the hypothesis that reinstatement of motifs led to better integration of A/B and C parts. Also, we observed a storyline x time effect (Fig. 3), which is consistent with the hypothesis that new segments updated the representations of the two storylines and pushed them further apart. It is worth noting that the storyline x time effect on its own does not provide definitive evidence for the within-storyline integration. For example, as the narrative continues, the understanding may be more similar across listeners, driving greater neural alignment across subjects.

After demonstrating the reinstatement of relevant past information, we next addressed the contribution of hippocampus-based episodic memory to this reinstatement. Previous work by Chen et al. (2016) showed that hippocampal-cortical interaction helped participants to integrate information across movie segments separated by a 1-day break. In our exploratory analyses, we found that storyline reinstatement increased with functional connectivity to hippocampus in mPFC both across subjects and across segments (Supplementary Fig. 13), for several (but not all) parameter settings for the analysis. This finding provides preliminary support for the idea that the hippocampus, which is known to be involved with the reinstatement of episodic memories over days, may also be involved with the reinstatement of recently accumulated memories over minutes. Notably, the degree of inter-subject functional connectivity between hippocampus and cortex did not reliably predict neural reinstatement triggered by motifs in part C. One possible explanation is that the inter-subject functional connectivity method involves averaging over multiple time points; this may make it less useful for detecting brief reinstatement events triggered by motifs.

In addition to hippocampus-mediated episodic memory, there are other mechanisms that might contribute to reinstating storyline representations. For example, recent studies of working memory have shown that past information could be held in the cortex during a delay period without persistent activity ^18–20 and such inactive neural patterns can be reactivated on task demand ²¹, by probe stimuli ¹⁹, and by transcranial magnetic stimulation ²²; several computational models have been built to account for the latent memory ²³ and short-term synaptic plasticity in cortex has been proposed to be the underlying mechanism^24–26. More research is needed to directly test whether short-term plasticity within cortex plays a role in supporting the memory of recent context.

Our design successfully induced reactivation of neural patterns associated with specific storylines and motifs, which, we believe, reflects the reinstatement of narrative information relating to past events rather than simple reactivation of word representations. First, we found a significant motif effect even when taking storyline-specific high-frequency words as the baseline (Fig. 6). Second, as noted above, we found a correlation between the behavioral relation scores and the neural motif effects (Fig. 7). In other words, the same set of narrative motifs yielded greater neural reinstatement in subjects who demonstrated a better understanding of the narrative. Third, the same motif was not always expressed in the same words ("throwing up” vs. “Clara feels sick, as the coffeecake rises to her throat”). As for the storyline effect, the strongest difference between storylines was found in regions with long TRW, i.e. regions where randomly ordered words failed to elicit reliable responses¹ (Fig. 2c). Furthermore, 27% of the word tokens in the early AB part are storyline specific, while only 24% of the word tokens in the late AB part are storyline specific. Therefore, the difference in wording could hardly explain the stronger storyline effect in the later AB part (Fig. 3).

In conclusion, real-life events require dynamic integration of past and present information. Our results suggest that process-memory may have two states, a state in which prior events are active and influence ongoing information processing, and an inactive state, in which the latent memory does not interfere with the ongoing neural dynamics ^6,18. Through cross-disciplinary collaboration, this study demonstrated a way to achieve some experimental control over naturalistic stimuli and showed how skilled storytellers leverage these mechanisms of separation and integration to bring about the desired effects in the listener’s brain.

Participants

Twenty-eight participants were recruited. They were all right-handed native English speakers. All participants provided written consent forms before the experiment. Twenty-five participants were included for further analyses (14 females, age 18-40). Three were excluded, one due to anatomical anomalies, one due to excessive motion artifacts in T1 image, and one slept during the story. The experimental protocol was approved by the Institutional Review Board of Princeton University.

Stimulus

The stimulus was created by Lazaridi (“The 21st Year” -- Excerpt, copyright 2019), who has been in collaboration with our lab for a number of years ²⁷. She has years of experience in practicing and evolving the technique of organizing the audience’s understanding, memory, and interpretation of a narrative through screenplay writing and professional screenplay development around the world ²⁸. Compared to other types of writing, the creation of a screenplay is highly audience-driven due to the large investment (in time, collaboration, and financing) inherent in film-making. Furthermore, watching a film is a more continuous experience than reading a book, requiring the screenwriter to guide and unite the audience’s understanding and overall response to the narrative without loss of focus or inner thought digressions.

Lazaridi designed the narrative stimulus as a stand-alone fiction text that incorporated her experience-guided narrative techniques of traditional screenplay writing. The narrative consisted of 45 segments, and two seemingly unrelated storylines, A and B. A and B segments were presented in an interleaved manner for the first 30 segments. In the last 15 segments (Part C), the two storylines merged into a unified narrative. Each segment lasted for 41-57 TRs (mean: 46 TRs = 70 sec). They were separated by silent pauses of 3-4 TRs. The narrative was recorded by a professional actress (June Stein), who is a native English speaker, and directed by Lazaridi to ensure that the actor’s interpretation matched the author’s intent. The recording is 56 minutes long (see Supplementary Table 1 for the transcription of the story).

In the A and B segments, the author incorporated unique narrative motifs, i.e., specific images/situations/phrases that recurred in part C (see Fig. 1 and Fig. 4 for a sample motif and Supplementary Table 2 for a list of all the sentences containing motifs). The recurrence of motifs in part C is designed to trigger the reinstatement of specific moments from part AB, in order to evolve their meanings and to integrate the two storylines.

In total, there were 28 different narrative motifs, occurring 58 times in the AB part, and 36 times in part C. The same narrative motif was not always realized with the same words. The main technique Lazaridi used to make motifs memorable was to embed them in emotionally heightened narrative moments. For example, at the beginning of the narrative, (Part A) Clara serves chili during a party in LA and her interactions with the dish (serving, eating, throwing up after eating it) map a series of seminal emotional moments in her personal narrative.

Procedure

The recording of the narrative was presented using MATLAB 2010 (MathWorks) and Psychtoolbox 3 ²⁹ through MRI-compatible insert earphones (Sensimetrics, Model S14). MRI-safe passive noise-canceling headphones were placed over the earbuds for noise reduction and safety. To remove the initial signal drift and the common response to stimulus onset, the narrative was preceded by a 14 TR long musical stimulus, which was unrelated to the narrative and excluded from fMRI analysis. Participants filled a questionnaire after the scanning, to evaluate their overall comprehension of the narrative and their ability to relate events in different parts of the story that shared the same motifs.

MRI acquisition

Subjects were scanned in a 3T full-body MRI scanner (Skyra, Siemens) with a 20-channel head coil. For functional scans, images were acquired using a T2*-weighted echo planar imaging (EPI) pulse sequence (repetition time (TR), 1500 ms; echo time (TE), 28 ms; flip angle, 64°), each volume comprising 27 slices of 4 mm thickness with 0 mm gap; slice acquisition order was interleaved. In-plane resolution was 3 × 3 mm² (field of view (FOV), 192 × 192 mm²). Anatomical images were acquired using a T1-weighted magnetization-prepared rapid-acquisition gradient echo (MPRAGE) pulse sequence (TR, 2300 ms; TE, 3.08 ms; flip angle 9°; 0.86 x 0.86 x 0.9 mm³ resolution; FOV, 220 x 220 mm²). To minimize head movement, subjects' heads were stabilized with foam padding.

MRI analysis

Preprocessing

MRI data were preprocessed using FSL 5.0 (http://fsl.fmrib.ox.ac.uk/) and NeuroPipe (https://github.com/ntblab/neuropipe), including BET brain extraction, slice time correction, motion correction, high-pass filtering (140 s cutoff), and spatial smoothing (FWHM 6 mm). All data were aligned to standard 3 mm MNI space (MNI152). Only voxels covered by all participants’ image acquisition area were included for further analysis.

Following preprocessing, the first 19 TRs were cropped to remove the music preceding the narrative (14 TRs), the time gap between scanning and narrative onset (2 TRs), and to correct for the hemodynamic delay (3 TRs). To verify the temporal alignment between the fMRI data and the stimulus, we computed the temporal correlation between the audio envelop of the stimulus (volume) and the subjects’ mean brain activation in left Heschl’s gyrus following Honey et al. ³⁰. The left Heschl’s gyrus mask was from Harvard-Oxford cortical structural probabilistic atlases (thresholded at 25%). The audio envelope was calculated using a Hilbert transform and down-sampled to the 1.5 s TR. The correlations were computed with -100-100 TRs lag to find the time lag that showed the highest correlation. The averaged peak time was 0.12 TR across subjects, indicating that the narrative and fMRI data were temporally well-aligned.

To account for the low-level properties of the stimulus, a multiple-regression model was built for each voxel. The regressors included an intercept, the audio envelope, and the boxcar function of the between-segment pauses, convolved by the canonical hemodynamic response function and its derivatives with respect to time and dispersion as given in SPM8 (https://www.fil.ion.ucl.ac.uk/spm/). For the effect of audio amplitude and between-segment pause, please see Supplementary Fig. 1. The residuals of the regression model were used for the following analyses.

ROI masks

We used 238 functional regions of interest (ROIs) defined independently by Shen et al.¹⁶ based on whole-brain parcellation of resting-state fMRI data. A control anatomical ROI was also included: left Heschl’s gyrus defined using the Harvard-Oxford cortical structural probabilistic atlas, thresholded at 25%.

A bilateral anatomical hippocampal mask was obtained using the same threshold. It has been proposed that the temporal integration window of episodic memory representation varies along hippocampal long axis ³¹. While the posterior hippocampus has a small representation scale, the middle and anterior portions are able to contain the associations between more than two events. If so, it would be inappropriate to treat hippocampus as a functionally homogenous region. Therefore, we divided the hippocampus mask into anterior (MNI coordinate y>-19), middle (-30 < y < =-19 ), and posterior parts (y <= -30) ROIs following Collin et al. (2015)³¹.

All ROIs had more than 50 voxels in our data.

Shared response model

When comparing activation patterns across subjects, the mismatch of functional topographies could decrease analysis sensitivity even after anatomical alignment ^32,33. Therefore, we functionally aligned data within each ROI across subjects using the shared response model (SRM)(Brain Imaging Analysis Kit, http://brainiak.org) ³⁴. SRM projects all subjects’ data into a common low-dimensional feature space by capturing the components of the response shared across subjects. The input to SRM was a TR x voxel x subject matrix, and the output was a TR x feature x subject matrix. We used fMRI data from the whole story (z-scored over time first) to estimate an SRM with 50 features. Note that no information about storyline or motif was submitted to SRM. Therefore, while this projection inflated the overall inter-subject pattern similarity, it could not artifactually give rise to the storyline or motif effect shown here. The output of SRM was z-scored over time. Unless otherwise stated, all the pattern analyses described below were run based on the resulting 50 features.

We also performed the same analyses without the application of SRM. Generally speaking, a subset of the areas that were significant in the analysis with SRM were also significant in the analysis without SRM. Please see Supplementary Fig. 2 for the results.

RSA of storyline effect

To examine the storyline effect, we performed representational similarity analysis (RSA) ^14,15 on brain activation patterns and tested whether the representational similarity between segments from the same storyline was higher than that of segments from different storylines. We first computed the averaged activation within each segment across TRs for each voxel. The resulting 45 values were then z-scored across segments. For each ROI, pairwise pattern similarities between the 45 activation maps were computed. Pairwise pattern similarities between the 45 activation maps were computed with the leave-one-subject-out method (Fig. 1b). Namely, the averaged activation pattern was extracted for each segment. Then the Pearson correlation coefficients between one subject’s activation patterns and the averaged patterns of the remaining subjects were computed. The output correlation coefficients (45 x 45 segments) were normalized with Fisher’s z-transformation. This procedure was repeated for each of the 25 subjects and each ROI.

We then contrasted the averaged within- and between-storyline similarities in the AB part, excluding the within-segment similarities (the diagonal of the 45 x 45 similarity matrix), to obtain 25 contrast values (Fig. 2a) for each ROI. These contrast values were compared to zero by a one-tailed one-sample t-test and thresholded at p < .05 (FWE correction for multiple comparisons).

To examine whether the storyline effect increased over time, for regions showing a significant storyline effect, we computed the storyline effect in the early (segment 1-14) and later (segment 15-30) halves of the AB part separately. 25 contrast values were generated by comparing the late and early storyline effects (late (same > different storyline) > early (same > different storyline)). These contrast values were again submitted to a one-tailed one-sample t-test (p < .05, FWE). The results were projected back onto the whole-brain surface and visualized using Freesurfer v6 (http://surfer.nmr.mgh.harvard.edu/).

To test the storyline x time effect in a more graded way, we constructed a 45 x 45 time effect matrix, populated with the average of the time points (segment number). For example, the (4, 5) entry of this matrix is 4.5 (=(4+5)/2). Taking only the entries corresponding to within-storyline similarity, excluding the diagonal elements (Supplementary Fig. 5, upper panel), we computed the Pearson correlation between the time matrix and the pattern similarity matrix. The resulting R-values were entered into a one-sample one-tailed group t-test after Fisher’s z-transformation within regions showing significant storyline effect (N=25, p < .05, FWE). The between-storyline dissimilarity was tested separately in a similar manner. The overlap between these two effects was shown in the lower panel of Supplementary Fig. 5.

Temporal receptive window index

Following Yeshurun et al. (2017)⁵, the TRW index was generated based on an independent dataset from Lerner et al. (2011)¹, which includes an intact story (“Pieman”, ~7 min long) and the same story with scrambled word order. Inter-subject correlation between averaged time series of each ROI was computed, using the leave-one-subject-out method, and normalized using Fisher’s z transformation. TRW index was then calculated by subtracting the ISC of the scrambled story from that of the intact story. We examined the correlation between TRW and storyline effect (Fig. 2c) across regions.

Time course of the storyline effect at segment boundary

To further illustrate the time course of the storyline effect, a long TRW ROI, i.e. posterior cingulate cortex, was selected. We computed the pattern similarities between each of the -40-40 TRs around segment onsets and the typical A and B storyline patterns using a leave-one-subject-out method. For example, for the boundary between segment 1 and segment 2, -40~40 TRs around the onset of segment 2 were extracted from one subject. The typical A storyline pattern was obtained by averaging all the A storyline TRs, except for the segments analyzed here, i.e. segment 1 and 2, from the rest of the subjects. The typical B storyline pattern was obtained in the same manner. Pearson correlation between the 81 TRs around segment 2 onset and the typical A and B patterns were calculated and normalized with Fisher’s z-transformation. The same procedure was repeated for each subject and each boundary. Fig. 2b shows the transition from B to A segments. Please see Supplementary Fig. 3 for the transition from A to B segments.

RSA of narrative motif effect

For each narrative motif occurrence, we obtained the corresponding activation pattern by averaging 5 TRs immediately after its onset based on the intuition that motif effect was transient and lasted only for a few sentences. Pearson correlation coefficients between activation patterns of motifs in the AB part and motifs in the C part were computed with the leave-one-subject-out method and normalized with Fisher’s z transformation. As shown in Fig. 4, pattern similarities between narrative motifs were grouped into three types: (1) same motif, (2) different motifs from the same storyline, and (3) different motifs from different storylines (unrelated). For example, pattern similarities between different occurrences of “chili” belong to (1). Similarities between “chili” and other A storyline motifs belong to (2). Similarities between chili and B storyline motifs belong to (3). Motif effect of each “chili” token in part C was defined as the averaged type (1) similarity minus the averaged type (2) similarity, in order to eliminate the confound of storyline effect.

The group motif effect was thresholded with a permutation test. For each ROI, the above procedure was repeated after shuffling the labels of motifs within storylines for 10000 times, creating a null distribution. To correct for multiple comparisons across ROIs, the largest motif effect across ROIs in each of the 10000 iterations was extracted, resulting in a null distribution of the maximum motif effects. Only ROIs with a group motif effect exceeding 95% of the null distribution were considered significant (p < .05, FWE)(Fig. 5, middle).

Time course of the narrative motif effect

To further illustrate the time course of the motif effect, for each motif in C, the Pearson correlation coefficients between activation patterns of -5~10 TRs around its onset and the activation patterns of motifs in the AB part were computed. Motifs with a -5~10 TRs time window that overlapped with the between-segment silent pauses were excluded from this analysis. The resulting coefficients were normalized with Fisher’s z-transformation and averaged by categories (same motif and same storyline, different motif but same storyline, and unrelated). For each ROI, we applied two-tailed paired t-tests to compare pattern similarities between categories at each time point (p < .05, FWE correction for time points). Fig. 5 shows the resulting pattern similarity around narrative motif onset.

Narrative motif vs. High-frequency word effect

To verify that the motif effect did not result from repeated wordings or word-level semantics, we replaced the narrative motifs with storyline-specific high-frequency words and performed the same RSA. More specifically, among words that only occurred in A and C parts and words that occurred only in B and C parts, we chose the 28 words with the highest lemma/word stem frequencies (Supplementary Table 4). Two out of the twenty-eight narrative motifs were included in this list. Together, these words occurred 111 times in the AB part and 110 times in the C part. Among regions showing a significant motif effect, we calculated the difference between the real motif effect and the effect elicited by high-frequency words for each subject. The 25 difference values were entered into a one-sample one-tailed t-test. The results were thresholded at p < .05 (FWE, Fig. 6).

Within Subject RSA of the storyline and motif effect

We used across-subject analyses to boost the signal-to-noise. In a prior study (Chen et al., 2017, Nature Neuroscience)¹⁷, we found that neural patterns associated with the perception and retrieval of specific events in a movie are shared across subjects. The finding of such shared neural coding indicates that averaging the neural patterns across subjects can boost the signal-to-noise. For a similar reasoning and detailed analysis of such effects, please see Simony et al. (2016)¹¹.

Although averaging responses across subjects is expected to improve our SNR, we are able to detect the same effect within individuals. As we predicted (based on our prior work), the results of these within-subject analyses are qualitatively similar to the across-subject analyses but somewhat weaker. We included the results of within-subject RSA thresholded using group t-test as in inter-subject RSA in Supplementary Fig. 6-9. Considering the potential impact of temporal autocorrelation³⁵ and low-frequency drift³⁶ in the fMRI signals on within-subject similarity matrix, especially between neighboring segments, we also included results thresholded using the label permutation method (Supplementary Fig. 10). For the storyline effect, we shuffled the labels of segment 1-30 10000 times to obtain a null distribution of the group mean effect. This procedure was performed for each ROI and the resulting p-value was corrected for multiple comparisons across ROIs (FWE). The storyline x time effect was tested within regions showing significant storyline effect by shuffling segment labels within storylines.

Correlation between hippocampal-cortical ISFC and cortical reinstatement of storyline and motif

To examine whether the cortical reinstatement of storyline was dependent on connectivity with hippocampus, we examined the Pearson correlation between hippocampal-cortical inter-subject connectivity (ISFC)¹¹ and the storyline effect across segments for each subject, within ROIs showing a significant storyline effect.

The ISFC was computed within the 0-40 TR time window after the onset of each segment using the leave-one-subject-out method, i.e. the correlation between one subject’s hippocampal activity and the averaged cortical activity of the other subjects. We used the preprocessed data without regressing out the effects of between-segment pause and the audio envelope because it is possible that the activation pulse between segments (Supplementary Fig. 1) does not only reflect the silence but also memory encoding or retrieval³⁷. SRM was not applied because topographical alignment is not a concern when comparing the averaged time series between ROIs. The hippocampus seed was defined using Harvard-Oxford cortical structural probabilistic atlas thresholded at 25%.

The storyline effect for each segment was also computed using the leave-one-subject-out method. The typical activation patterns for A and B storylines were first estimated by averaging data from all but one subject, excluding the current segment. Pattern similarity between the resulting typical A & B patterns and the left-out subject’s activation pattern for the current segment was then computed. The storyline effect was defined as the difference between pattern similarity to the relevant storyline and the similarity to the irrelevant storyline, taking the previous segment as the baseline, for example, for a B segment: (current segment’s similarity to B - similarity to A) - (previous segment’s similarity to B - similarity to A).

The correlation between ISFC and the storyline effect across segments was computed for each subject, excluding the first segment of each storyline. The R-values were entered into a one-tailed t-test after Fisher’s z-transformation. This initial analysis did not yield a significant result after correction for multiple ROIs (N = 25, p < .05, FDR correction). In exploratory follow-up analyses, we then systematically examined the influence of the time window of ISFC, the time window of the storyline effect, the hippocampus seed (whole vs. posterior: MNI y <= -30), and the baseline of the storyline effect. We also examined the correlation across subjects in each segment. For each combination of analysis parameters, we corrected for multiple-comparisons across ROIs using the FDR method. Please see Supplementary Fig. 13 for the results. No significant correlation was found after simultaneous FDR-correction for multiple-comparisons across ROIs and the twelve sets of analysis parameters.

We examined the correlation between hippocampal ISFC and motif reinstatement in a similar manner. The motif effect was defined and computed using the RSA method described above (based on 5 TRs after motif onsets, using similarity between different motifs of the same storylines as baseline)(Fig. 4). Across motifs in the C part, correlation between the motif effect and ISFC after motif onset was then computed for each subject. We also examined the correlation across subjects for each motif and the influence of ISFC time windows and hippocampus seeds.

Data availability. The data used in this study have been publicly released as part of the "Narratives" collection. Raw MRI data are formatted according to the Brain Imaging Data Structure (BIDS) with exhaustive metadata and are publicly available on OpenNeuro: https://openneuro.org/datasets/ds002245. The data corresponding to this study are indicated using the "21styear" task label. These data can be cited using the following reference:

Nastase, S. A., Liu, Y.-F., Hillman, H., Zadbood, A., Hasenfratz, L., Keshavarzian, N., Chen, J., Honey, C. J., Yeshurun, Y., Regev, M., Nguyen, M., Chang, C. H. C., Baldassano, C. B., Lositsky, O., Simony, E., Chow, M. A., Leong, Y. C., Brooks, P. P., Micciche, E., Choe, G., Goldstein, A., Halchenko, Y. O., Norman, K. A., & Hasson, U. Narratives: fMRI data for evaluating models of naturalistic language comprehension. https://doi.org/10.18112/openneuro.ds002245.v1.0.3

Acknowledgment

This study is supported by a Magic Grant from Princeton’s Humanities Council, Taiwan Ministry of Science and Technology (105-2917-I-564 -007 -), and by the National Institute of Mental Health (R01-MH112357).

Author contributions

C.L., Y.Y., and U.H. designed the experiment; C.L authored the stimulus and C.L and Y.Y created the audio rendition of the narrative. C.H.C. and Y.Y. acquired the data; C.H.C. analyzed and interpreted the data with input from U.H. C.L. and K.A.N.; C.H.C. drafted the manuscript; U.H., K.A.N., Y.Y., and C.L. substantively revised the manuscript.

Additional information

Competing financial interests: The authors declare no competing financial interests.

Lerner, Y., Honey, C. J., Silbert, L. J. &Hasson, U. Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. J. Neurosci. 31, 2906–2915 (2011).
Honey, C. J. et al. Slow Cortical Dynamics and the Accumulation of Information over Long Timescales. Neuron 76, 423–434 (2012).
Chen, J. et al. Accessing Real-Life Episodic Information from Minutes versus Hours Earlier Modulates Hippocampal and High-Order Cortical Dynamics. Cereb. Cortex 26, 3428–3441 (2016).
Baldassano, C. et al. Discovering Event Structure in Continuous Narrative Perception and Memory. Neuron 95, 709-721.e5 (2017).
Yeshurun, Y., Nguyen, M. &Hasson, U. Amplification of local changes along the timescale processing hierarchy. Proc. Natl. Acad. Sci. 201701652 (2017). doi:10.1073/pnas.1701652114
Hasson, U., Chen, J. &Honey, C. J. Hierarchical process memory: Memory as an integral component of information processing. Trends Cogn. Sci. 19, 304–313 (2015).
Cowan, N. Chapter 20 What are the differences between long-term, short-term, and working memory? Prog. Brain Res. 169, 323–338 (2008).
Baddeley, A. Working memory: looking back and looking forward. Nat. Rev. Neurosci. 4, 829–839 (2003).
Raichle, M. E. et al. A default mode of brain function. Proc. Natl. Acad. Sci. U. S. A. 98, 676–82 (2001).
Buckner, R. L., Andrews-Hanna, J. R. &Schacter, D. L. The brain’s default network: Anatomy, function, and relevance to disease. Ann. N. Y. Acad. Sci. 1124, 1–38 (2008).
Simony, E. et al. Dynamic reconfiguration of the default mode network during narrative comprehension. Nat. Commun. 7, 12141 (2016).
Lahnakoski, J. M., Jääskeläinen, I. P., Sams, M. &Nummenmaa, L. Neural mechanisms for integrating consecutive and interleaved natural events. Hum. Brain Mapp. 00, (2017).
Milivojevic, B. et al. Coding of Event Nodes and Narrative Context in the Hippocampus. J. Neurosci. 36, 12412–12424 (2016).
Kriegeskorte, N., Goebel, R. &Bandettini, P. Information-based functional brain mapping. Proc. Natl. Acad. Sci. U. S. A. 103, 3863–8 (2006).
Kriegeskorte, N. Representational similarity analysis – connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).
Shen, X., Tokoglu, F., Papademetris, X. &Constable, R. T. Groupwise whole-brain parcellation from resting-state fMRI data for network node identification. Neuroimage 82, 403–415 (2013).
Chen, J. et al. Shared memories reveal shared structure in neural activity across individuals. Nature Neuroscience 20, (NIH Public Access, 2017).
Stokes, M. G. ‘Activity-silent’ working memory in prefrontal cortex: a dynamic coding framework. Trends Cogn. Sci. 19, 394–405 (2015).
Wolff, M. J., Jochim, J., Akyürek, E. G. &Stokes, M. G. Dynamic hidden states underlying working-memory-guided behavior. Nat. Neurosci. 20, 864–871 (2017).
Sprague, T. C., Ester, E. F. &Serences, J. T. Restoring Latent Visual Working Memory Representations in Human Cortex. Neuron 91, 694–707 (2016).
Watanabe, K. &Funahashi, S. Neural mechanisms of dual-task interference and cognitive capacity limitation in the prefrontal cortex. Nat. Neurosci. 17, 601–611 (2014).
Rose, N. S. et al. Reactivation of latent working memories with transcranial magnetic stimulation. Science (80-. ). 354, 1136–1139 (2016).
Buonomano, D.V. &Maass, W. State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113–125 (2009).
Miller, E. K., Lundqvist, M. &Bastos, A. M. Working Memory 2.0. Neuron 100, 463–475 (2018).
Zucker, R. S. &Regehr, W. G. Short-Term Synaptic Plasticity. Annu. Rev. Physiol. 64, 355–405 (2002).
Mongillo, G., Barak, O. &Tsodyks, M. SynaptiC Theory of Working Memory. Science (80-. ). 319, 1543–1546 (2008).
Yeshurun, Y. et al. Same Story, Different Story: The Neural Representation of Interpretive Frameworks. Psychol. Sci. 28, 307–319 (2017).
Lazaridi, C. Stories that Change: A Diagnostic Manual for Troubleshooting Your Screenplay. (Mediterranean Film Institute, 2012).
Brainard, D. H. The Psychophysics Toolbox. Spat. Vis. 10, 433–436 (1997).
Honey, C. J., Thompson, C. R., Lerner, Y. &Hasson, U. Not Lost in Translation: Neural Responses Shared Across Languages. J. Neurosci. 32, 15277–15283 (2012).
Collin, S. H. P., Milivojevic, B. &Doeller, C. F. Memory hierarchies map onto the hippocampal long axis in humans. Nat. Neurosci. 18, 1562–1564 (2015).
Brett, M., Johnsrude, I. S. &Owen, A. M. The problem of functional localization in the human brain. Nat. Rev. Neurosci. 3, 243–249 (2002).
Sabuncu, M. R. et al. Function-based Intersubject Alignment of Human Cortical Anatomy. Cereb. Cortex 20, 130–140 (2010).
Chen, P.-H. et al. A Reduced-Dimension fMRI Shared Response Model. Neural Inf. Process. Syst. Conf. 460–468 (2015).
Mumford, J. A., Davis, T. &Poldrack, R. A. The impact of study design on pattern estimation for single-trial multivariate pattern analysis. Neuroimage 103, 130–138 (2014).
Alink, A., Walther, A., Krugliak, A., van denBosch, J. J. F. &Kriegeskorte, N. Mind the drift - improving sensitivity to fMRI pattern information by accounting for temporal pattern drift. bioRxiv 032391 (2015). doi:10.1101/032391
Ben-Yakov, A. &Dudai, Y. Constructing realistic engrams: poststimulus activity of hippocampus and dorsal striatum predicts subsequent episodic memory. J. Neurosci. 31, 9032–42 (2011).

There is NO Competing Interest.

ABCSupplementaryFigures200820.pdf
Supplementary Figures
SupplementaryTable1.xlsx
Supplementary Table 1
SupplementaryTable2.xlsx
Supplementary Table 2
SupplementaryTable45.pdf
Supplementary Table 4-5
SupplementaryTable3.xlsx
Supplementary Table 3

Download PDF

Journal Publication

published 01 May, 2021

Read the published version in Journal of Cognitive Neuroscience →

Version 1

posted

You are reading this latest preprint version

Relating the past with the present: Information integration and segregation during ongoing narrative processing

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Results

Discussion

Methods

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1