We used multiple neuroimaging techniques to ask whether the principles of dynamic hierarchical predictive coding can explain the location and timing of evoked neural activity produced by expected, unexpected and anomalous words during language comprehension. We showed that, relative to predicted continuations, words carrying unpredicted lexico-semantic information produced larger evoked responses at lower levels of the left fronto-temporal language hierarchy (left temporal cortex), while words that additionally violated higher-order contextual constraints produced activity at higher levels of the hierarchy (left inferior frontal cortex). In a later time window, prediction violations also activated different parts of the temporal cortex depending on whether they resulted in plausible or anomalous interpretations. We first describe the pattern of MEG and ERP effects for each contrast of interest. We then turn to the pattern of activity revealed by fMRI across the four conditions, discussing both its divergence and convergence with the source-localized MEG effects.
Lower-level lexico-semantic prediction error within left temporal cortex is produced by incoming words, regardless of contextual constraint
Consistent with many previous ERP studies20–22, contextually unexpected words produced a larger N400 between 300-500ms at the scalp surface than expected words. A key claim of predictive coding is that differences in evoked activity between expected and unexpected inputs are driven by the top-down suppression of prediction error to expected inputs at lower levels of the cortical hierarchy (expectation suppression6,7). Our MEG findings support this claim. The evoked effect between 300-500ms localized to multiple regions within left temporal cortex that are known to support lexical and semantic processing. These included left anterior temporal cortices (ventral and superior/middle temporal), which function to “bind” widely distributed semantic features into distinct concepts28, and left mid- temporal cortices (mid-superior/middle temporal29,30 and mid-fusiform31), which function to map orthographic and phonological representations onto meaning (lexical processing).
Previous MEG32 and intracranial studies33 have also reported increased activation in temporal cortex to unexpected (versus expected) words in the N400 time window. However, in these earlier studies, the unexpected words were often implausible or they violated strong contextual constraints. Using plausible sentences, we showed that, between 300-500ms, the activity evoked by unexpected words within the temporal cortex was very similar in low constraint and high constraint contexts. This provides strong evidence that, instead of reflecting an enhanced response to implausible continuations, or the costs of inhibiting incorrect lexico-semantic predictions, these differences were driven by the top-down facilitation of expected lexico-semantic information within the temporal cortex. Specifically, we suggest that, in high constraint contexts, comprehenders incrementally built an event model14 that generated top-down lexico-semantic reconstructions of expected upcoming words. These reconstructions immediately suppressed the lexico-semantic prediction error produced by new expected inputs.
In addition to these expectation suppression effects within left anterior and mid-temporal cortices, we also observed an MEG effect in the left medial temporal cortex within the same 300-500ms time window, consistent with previous intracranial studies33. This medial temporal effect, however, was not only driven by a dipole to the unexpected critical words, but also by a dipole in the opposite direction to the expected critical words. We suggest that the dipole to the unexpected words reflected a functional role of the left medial temporal cortex (along with anterior lateral temporal regions) in retrieving and binding the semantic features associated with the incoming word28, possibly supported by “pattern completion” within the hippocampus itself27. The dipole to the expected words may have reflected a neural “resonance”34 within medial temporal subpopulations that were already pre-activated prior to encountering the new bottom-up input35. The presence of two dipoles going in opposite directions may explain why previous MEG studies have failed to detect effects within the medial temporal cortex within the N400 time window. This is because most MEG studies have used unsigned, rather than signed, dipole values for source localization, and the absolute values of two dipoles going in opposite directions are likely to cancel out.
Higher-level prediction error within left inferior frontal cortex is produced only by words that violate high certainty predictions
A key assumption of the account outlined above is that the top-down lexico-semantic reconstructions that suppress lower-level prediction error are informed by long-term schema knowledge that is relevant to the current message being communicated. Within this hierarchical framework, these schemas are represented at the highest level of the generative hierarchy, and they themselves generate reconstructions that constrain the current event model15. During real-world language comprehension, however, messages can change rapidly. In order to continue predicting effectively, comprehenders must be able to recognize event boundaries36 so that they can rapidly shift the event model by retrieving new high-level schemas15,18. Dynamic hierarchical predictive coding makes two important claims regarding these high-level shifts. First, they are triggered by higher-level prediction error, which is produced whenever new inputs violate a high confidence prior belief in the higher-level state11. Second, they result in the generation of new top-down reconstructions that provide retroactive feedback to lower levels of the cortical hierarchy, enhancing activity over consistent representations (top-down “sharpening”4,13).
Our findings support both these claims. Replicating previous ERP studies21,22, we found that, relative to expected words, unexpected words produced a late frontal positivity ERP effect between 600-1000ms only in high constraint contexts. In MEG, the same contrast revealed activity within the left inferior frontal cortex in this late time window. This was accompanied by a re-activation of the left middle temporal cortex. No late frontal or temporal effects were observed when contrasting expected words with unexpected words in low constraint contexts.
We suggest that in both the low and high constraint contexts, the lower-level lexico-semantic prediction error led comprehenders to infer a new plausible event, resulting in the production of reconstructions that switched off the lower-level lexico-semantic prediction error, thereby attenuating the evoked response within the left temporal cortex at the end of the N400 time window. However, in the high constraint context, this newly inferred event violated a prior high-certainty belief in a different event that had previously been inferred from the context37,38. This increased the gain on the new event information, resulting in a higher-level event prediction error within the left inferior frontal cortex in the later 600-1000ms time window. This higher-level prediction error initiated the retrieval of a new schema from long-term memory18, enabling comprephenders to successfully shift their event model, and resolve the error22,39. The updated event model, in turn, provided retroactive feedback to the left temporal cortex, enhancing activity over schema-consistent lexico-semantic representations, while reducing activity over incorrectly predicted lexico-semantic information15. The top-down nature of this feedback enhancement may explain why, within this late time window, the dipoles within the temporal cortex were of the opposite polarity to those produced by the bottom-up prediction error within the 300-500ms time window. This account is also consistent with the well-known role of the left inferior frontal cortex in top-down suppression and selection40.
A breakdown of predictive coding to anomalous words
This hierarchical predictive coding framework posits that higher-level prediction error should also be produced if a newly updated state is inconsistent with prior reconstructions received from a still higher cortical level. Critically, however, if this higher-level prediction error cannot be resolved because the input is incompatible with the constraints of the generative model, or with alternative models stored in long-term memory, then the late retrieval and top-down sharpening mechanisms described above should break down. For example, after encountering a semantic anomaly, it is impossible to retrieve a new schema that can explain the input, and so the conflict between the top-down reconstructions produced by the current schema and the bottom-up lexico-semantic prediction error cannot be resolved. This will therefore lead to (a) a failure to switch off prediction error at even lower levels of the cortical hierarchy (perceptual reanalysis), and/or (b) new learning in order to explain the input18,19.
Our findings are broadly consistent with this account. First, at the scalp surface, the anomalous words produced an N400 that was larger than that produced by the plausible unexpected continuations (this difference was less prominent in ERP than in MEG, see Supplementary Materials Sect. 3). MEG localized the activity within this 300-500ms time window not only to the left temporal cortex, but also to the left inferior frontal and anterior cingulate cortices. We suggest that the inferior frontal activity reflected the production of an early event prediction error (because the impossible event fell outside the range of event reconstructions generated by the current schema), and that the enhanced activity within the temporal cortex resulted from a failure to settle on a higher-level interpretation within this time window, and therefore to switch off lower-level lexico-semantic prediction error. The surprising failure to minimize prediction error within the N400 time window may have led to the early recruitment of the anterior cingulate cortex41.
Second, within the late time window (600-1000ms), the semantic anomalies also produced a late posterior positivity/P600 ERP effect, which is often triggered by high-level linguistic conflict22,25,26, and thought to reflect a lower-level reanalysis of the input22,25,39. Consistent with this proposal, in MEG we observed sustained activity within posterior fusiform cortex, which supports sub-lexical orthographic processing9. We suggest that this “orthographic reanalysis” arose because the brain failed to settle on a single lexico-semantic representation, and therefore failed to produce reconstructions that switched off orthographic prediction error at this still lower level of the linguistic hierarchy.
Finally, throughout the 600-1000ms window, semantic anomalies also produced an effect within the medial temporal cortex. This region is highly interconnected with the hippocampus, which plays a major role in detecting associative and contextual novelty42, primarily through a “comparator function” that tracks the magnitude of prediction violations43, thereby paving the way towards new learning44. Consequently, this medial temporal activation may have indirectly supported updates in the parameters of the generative model that allowed comprehenders to adapt to anomalous inputs (consistent with known links between the late posterior positivity/P600 and adaptation45). Alternatively, it may have supported the learning of new schemas from the novel anomalous inputs18,27,46. Both of these interpretations are consistent with the important computational role of prediction error in bridging comprehension and learning38.
Convergence and divergence between fMRI and MEG/EEG
A second goal of this study was to understand how hemodynamic activity, recorded using fMRI, converged and diverged from the pattern of ERP and source-localized MEG effects produced in the same paradigm and in the same group of participants.
The clearest discrepancy between the fMRI and MEG/EEG data was that fMRI failed to detect the ERP and MEG effects observed in the N400 time window (300-500ms). For example, even though the contrast between the low constraint unexpected and expected critical words revealed significant MEG effects within left lateral, ventral and medial temporal cortices (corresponding to the N400 effect), the same contrast in fMRI showed no significant differences within the temporal cortex. The contrast between high constraint unexpected and expected critical words did reveal some hemodynamic activity within the left middle temporal cortex, and the contrast between anomalous and expected words revealed activity within the fusiform cortex. However, both these effects can be explained by later MEG/EEG activity, from 600-1000ms.
Although striking, this insensitivity of the hemodynamic response to N400 activity is not altogether surprising. Others have noted that MEG is more likely to localize top-down contextual effects to the temporal lobe than fMRI30. In addition, multimodal neuroimaging studies of semantic priming report fMRI effects that are much smaller and less robust than MEG N400 effects47,48. A likely reason for these discrepancies is that, while MEG and EEG are highly sensitive to brief, time-locked activity49, fMRI is relatively blind to transient responses that are associated with the initial stages of feedforward activity50,51.
Conversely, because the hemodynamic response integrates activity across multiple successive time windows, the signal is dominated by activity at later stages of processing. Indeed, the clearest pattern of convergence between fMRI effects and source-localized MEG effects was within the 600-1000ms time window. Both techniques revealed effects within the left frontal/middle temporal cortex to high constraint unexpected (versus expected) critical words, and within the left frontal/fusiform cortex to anomalous (versus expected) critical words. Consistent with previous studies50,52, activity within the prefrontal cortex was more robust and extensive in fMRI than MEG (note that the left frontal effect to low constraint unexpected versus expected critical words was significant in fMRI but not in MEG). This may be because MEG is insensitive to radial sources from gyri, and because tangential sources on opposing sides of sulci can cancel out53. It is also possible that the hemodynamic response was less time-locked to the critical words, and that it detected activity past 1000ms. Nonetheless, given the challenges of solving the inverse problem, the qualitative similarity between the MEG activity detected within the late time window and the hemodynamic response in the same contrasts provides independent corroborating evidence for the late MEG source-localized effects.