Complementary hemispheric lateralization of language and social processing in the human brain

Humans have a unique ability to use language for social communication. The neural architecture for language comprehension and production may have prominently emerged in the brain areas that were originally involved in social cognition. Here, we directly tested the fundamental link between language and social processing using functional magnetic resonance data (MRI) data from over 1,000 human subjects. Cortical activations in language and social tasks showed a striking similarity with a complementary hemispheric lateralization. Within core language areas, left-lateralized activations in the language task were mirrored by right-lateralized activations in the social task. Outside these areas, the activations were left lateralized in both tasks, perhaps indicating multimodal integration of social and semantic information. Our ﬁndings could have important implications in understanding neurocognitive mechanisms of social disorders such as autism. predictor covered the duration of a single video clip (20 s). The regressors in the social task were deﬁned based on the presented stimuli (not the subjects’ responses). All regressors were convolved with a canonical hemody-namic response function and its temporal derivatives. The time-series were temporally ﬁltered with a Gaussian-weighted linear high-pass ﬁlter with a cutoff of 200 s, to remove low-frequency drifts/ﬂuctuations presumably unrelated to the task design. The time-series were also prewhitened to remove temporal autocorrelations in the fMRI signal. For three comparisons (‘story vs. baseline’, ‘story vs. math’, and ‘social vs. random’), the contrast of parameter estimate (COPE) was computed based on beta values of the GLM. The language task did not have typical ﬁxation blocks (there was only an 8-second ‘get ready’ countdown at the beginning of the run), and the baseline condition in this task represented the mean activity across all time-points in each run. Fixed-effects analyses were conducted to estimate the average effects across runs within each subject, then mixed-effects analyses treating subjects as random effects were conducted to obtain group-average maps. The group-average maps were Cohen’s d effect size maps. Cohen’s d in each vertex/voxel was computed as the mean COPE (across subjects) divided by the standard deviation. For thresholding, vertices with positive Cohen’s d values were ﬁrst selected, then the mean value across these vertices was used as the threshold. After binarizing, a multi-modal parcellation of cerebral cortex (Glasser et al., 2016b) was overlaid on the thresholded map to identify the auditory cortex (low-level and high-level auditory regions), divide the superior temporal region, and name activated language areas.

people (theory of mind) (8). Understanding the social content of the environment is an evolutionarily old skill, seen in many forms across the animal kingdom. For instance, brain imaging in macaque monkeys has shown a dedicated network of bilateral cortical areas in temporal and prefrontal cortex for processing social interactions (9,10). One specific form of social communication is linguistic communication, and the cortical architecture for language processing may have emerged in LH within territories that were primarily involved in social processing. Thus, based on this scenario, it would be expected to find that the homotopic areas in RH contain an original selectivity for social processing.

Results:
To investigate the relationship between cortical networks of language and social processing, we used fMRI data of 1044 healthy young adults from the Human Connectome Project (HCP) database (https://www.humanconnectome.org/study/hcp-young-adult). In each subject, cortical activation maps were obtained for language (11) and social (12,13) tasks. The language task consisted of story blocks and math blocks. In the story blocks, subjects were presented with brief auditory stories and answered questions about the topic of stories. In the math blocks, subjects were auditorily presented with simple arithmetic questions. In the social task, subjects were presented with short video clips of objects (squares, circles, triangles) that either interacted in some way (approaching, chasing, etc.) or moved randomly on the screen. Subjects judged whether the objects had a social interaction or not. Functional maps from all subjects were multimodally transformed to a standard cortical surface where LH and RH were precisely registered to each other (i.e. there was a one-to-one correspondence between points/vertices of LH and RH) (14). Having such correspondence was crucial for an anatomically accurate evaluation of inter-hemispheric relations over the entire cortex.
Group-average activation maps (effect size maps) for language and social tasks were obtained by statistically comparing the 'story vs. baseline' and 'social vs. random' conditions, respectively (Figure   1a,b). The stimuli and task instructions of the two tasks were drastically different. However, the activation maps showed a remarkable similarity in the topography of areas activated by story and social stimuli, especially in lateral temporal and lateral prefrontal cortex. It was also evident that the languagerelated activations were relatively stronger in LH, while the social-related activations were relatively stronger in RH. To perform a detailed analysis of hemispheric lateralization within core language areas, these areas were first localized based on the group-average language map in LH (Figure 1c and Supplementary Figure 1). The localized areas, named based on a multi-modal parcellation of cerebral cortex (15), included PGi, PSL (perisylvian language area), STSp (superior temporal sulcus posterior), STSa (superior temporal sulcus anterior), STGa (superior temporal gyrus anterior), SFL (superior frontal language area), 55b, and Broca's area (areas 44, 45, and 47) (Figure 1d). PSL, which was located within the planum temporale, could be considered part of Wernicke's area. Most of these areas were also activated in the 'story vs. math' contrastthough the separation between language and default mode networks was more pronounced in the 'story vs. baseline' contrast (Supplementary Figure 2). Previous work confirms that the language network identified by the 'story vs. baseline' contrast includes neurobiologically relevant and functionally coupled areas (15).
Next, we examined the hemispheric lateralization within core language areas. For each task and each subject, the activation map in RH was subtracted from the activation map in LH, then the difference maps were averaged across subjects. The difference maps would allow evaluation of lateralization regardless of idiosyncratic differences in absolute activation levels. As expected, the language areas showed LH dominance in the language task (Figure 1e). Interestingly, the same areas in RH showed a relatively higher activation to social stimuli (Figure 1f). To quantify the effects, the functional asymmetry values (LH activation minus RH activation) were averaged across vertices in each area. All areas except PGi showed a significant double-dissociation effect: LH dominance in the language task and RH dominance in the social task (Figure 2a).
Based on data of individual subjects, we selected a minority group of atypical subjects (n = 29) who clearly had strong language activations in RH (Figure 2b,c), and tested whether the RH laterality effect in the social task is reduced (or perhaps eliminated) in these subjects. In the group-average maps of these subjects, most of the areas included subregions showing LH dominance in the social task (Figure 2d).
Importantly at the areal level, none of the areas showed RH dominance in the social task (Supplementary Figure 3) contrary to what we observed in the group-average map of all subjects.
At the areal level, the pattern of hemispheric biases was generally consistent across areas. However, a detailed look at the fine-scale spatial organization of language and social lateralization revealed an important difference between STSp and other areas (Figure 3). Within STSp, there was a considerable overlap between subregions showing laterality effects in language and social tasks, whereas in other areas especially within PGi, these subregions actually avoided each other (Figure 3a,b). In STSp, common neural populations appear to be activated by language and social tasks (see also 16), and therefore, the opposite hemispheric lateralization for language and social processing may be accentuated in this area through competition for shared neural resources. This prediction was confirmed in a subsequent analysis where we systematically tested the relationship between functional asymmetry values of language and social processing across subjects (Figure 3c). STSp showed a highly significant negative correlation between language and social lateralization, meaning that subjects with stronger language activation in left STSp tended to have stronger social activation in right STSp. This negative correlation in STSp was observed in both right-handed and left-handed subjects (Supplementary Figure 4). PSL also showed a weak but significant negative correlation, while PGi showed a significant positive correlation.
In other areas, there was no association between language and social lateralization.
Do the individual differences in complementary hemispheric lateralization correlate with performance in language and social tasks? We addressed this question using behavioral data of all subjects in HCP. The behavioral measures of language processing were based on scores in vocabulary comprehension and reading recognition tests (17). The behavioral measure of social processing was based on accuracy in the social task during functional imaging. For two groups of subjects with high and low performance in these behavioral metrics, we estimated language and social lateralization effects in eight language areas (Supplementary Figure 5). In PGi and PSL, higher LH lateralization in the language task was linked to better performance in the language tests. In STSp and STSa, higher RH lateralization in the social task was linked to better accuracy in this task. Thus, at least in some areas, there was an association between hemispheric lateralization and behavioral outcome.
The asymmetry maps shown in Figure 1e,f also revealed lateralization effects outside the language network. To quantitatively evaluate the effects in all parts of cortex, we first computed functional asymmetries in the language task for all 180 cortical areas from multi-modal parcellation (15). In this parcellation, each parcel in LH had a corresponding parcel in RH. 25 parcels with the highest absolute values of asymmetry were then selected as regions-of-interest (ROIs) (Supplementary Figure 6) and visualized on the cortical surface (Figure 4a). We identified eight ROIs outside language-auditory network. When comparing with resting-state networks (18), these ROIs were predominantly located within default and frontoparietal networks (Supplementary Figure 7a). Although responses in these ROIs were generally negative, they could differentiate story and math stimuli (Supplementary Figure   7b). Interestingly, four frontal ROIs located immediately anterior to 55b and Broca's area showed LH dominance in both language and social tasks (Figure 4b,c). Furthermore, in the majority of non-language ROIs, degree of lateralization in the language task was positively correlated with degree of lateralization in the social task (Figure 4d and Supplementary Figure 8). A possible interpretation is that some LH regions outside the language network might be involved in integrating linguistic and social information.

Discussion:
The two cerebral hemispheres sometimes show complementarity of function. While LH is more specialized in fine-grained processing of visual stimuli, RH is more specialized in holistic processing of global form (19). In the case of category-selective areas of visual cortex, the fusiform face area (FFA) is generally right-lateralized (20,21), whereas the visual word form area (VWFA) is better localized in LH (22,23). The brain networks of language and attention also show opposite hemispheric lateralization (24). While the language areas are typically located in LH, RH appears to be specifically involved in the control of visuospatial attention. In all cases described above, the lateralization of a certain area/processing in one hemisphere is often accompanied with the lateralization of another area/processing in the other hemisphere. Complementary lateralization of language and social processing reported in our study is unique in the sense that the homologous areas of the two hemispheres showed such lateralization effects.
Previous studies have shown that the homologues of language areas in RH are involved in the processing of prosody (25). Affective-prosodic components of speech play an important role in social communications, and they may tightly linked to other elements of social processing in RH. It has also been reported that the homologue of Broca's area (Brodmann areas 44/45) in RH is active during the observation of hand/mouth actions performed by other individuals (26). Thus, this area along with other areas of the 'mirror-neuron' system could encode information necessary for gestural communication (27). Another possibility is that this area represents social contents of actions rather than actions per se.
Overall, the previously proposed functions for the RH homologues of language areas could be characterized as various forms of social processing.
It has been suggested that hemispheric lateralization has computational benefits (28). Unilateral systems could be efficient for coordinating rapid sequences of precise, ordered operations. In fact, the degree of functional asymmetry across the two hemispheres predicts behavioral measures of verbal and visuospatial ability (29). Such systems would also contribute to compact wiring of the brain by minimizing the aggregate length of cortico-cortical connections. Complementary hemispheric lateralization of language and social processing might be essential for a normal behavior in these domains, and it may be disrupted in autistic patients who have difficulty in both linguistic and social communications. A recent study has reported reduced language lateralization in autism (30)   (e,f) Functional asymmetry maps for each task were obtained by subtracting RH activations from LH activations.
The dataset included 1045 subjects. One subject (subject ID: 765864) had a large MR artifact, and this subject was excluded after consulting with the HCP team. The remaining 1044 subjects (559 females, 485 males) had complete functional data for both language and social tasks. Subjects were recruited from Washington University (St. Louis, MO) and the surrounding area. The HCP data were acquired using protocols approved by the Washington University institutional review board, and written informed consent was obtained from all subjects.

Data acquisition:
The HCP MRI data acquisition has previously been described in detail (Glasser et  coil. At least one 3D T1w MPRAGE image and one 3D T2w SPACE image were acquired at 0.7 mm isotropic resolution. Whole-brain resting-state fMRI and task fMRI data were acquired using multi-band EPI sequence with parameters of TR=720 ms, 2 mm isotropic voxels, and multi-band acceleration factor of 8. Spin echo field maps were acquired during both structural and fMRI scanning sessions to enable accurate cross-modal registration of structural and functional images in each subject.

Task paradigm:
Functional data in this study were based on the HCP language processing and the HCP social cognition (theory of mind) tasks (Barch et al., 2013).
The language task consisted of two runs (run duration = 3:57 min:sec). In each run, 4 blocks of a story task were interleaved with 4 blocks of a math task. The lengths of blocks varied, and the average duration of blocks was approximately 30 s. In the story blocks, participants were presented with brief auditory stories (5-9 sentences) adapted from Aesop's fables, followed by a 2-alternative forced-choice question that asked participants about the topic of the story. For example, after a story about an eagle that saves a man who had done him a favor, participants were asked "That was about revenge or reciprocity?" Participants pressed a button to select either the first or the second choice. The math task also included trials that were presented auditorily. In these trials, participants completed a series of simple arithmetic (addition and subtraction) operations (e.g., "Fourteen plus twelve"), followed by "equals" and then two choices (e.g., "twenty-nine or twenty-six"). Participants pressed a button to select either the first or the second answer. The math task was adaptive to maintain a similar level of difficulty across participants.
In the social task, participants were presented with short video clips (20 s) of objects (squares, circles, triangles) either interacting in some way or moving randomly. After each video clip, participants were required to choose between 3 possibilities: a social/mental interaction (an interaction that appears as if the shapes are taking into account each other's feelings and thoughts), not sure, or no interaction (i.e., there is no obvious interaction between the shapes, and the movement appears to be random). The social task consisted of two runs (run duration = 3:27 min:sec). Each run contained 5 video blocks (2 Social and 3 Random in one run, 3 Social and 2 Random in the other run) and 5 fixation blocks (15 s each).
The stimuli used in the social task had some low-level retinotopic differences, which could result in differential activations in early visual cortex. To test this possibility, we examined the root mean square (RMS) contrast of images in four quadrants of visual field. We found differences between quadrants (Supplementary Figure 9), which could potentially explain the apparent asymmetry patterns seen in early visual cortex of the social asymmetry map (Figure 1f). Thus, these patterns do not appear to be related to a genuine functional asymmetry for social processing.

Data analysis software:
Data were preprocessed and analyzed using the publicly released HCP pipelines (Glasser et al., 2013).

Analysis of structural data:
Structural images (T1w and T2w) were used for extracting subcortical gray matter structures and reconstructing cortical surfaces in each subject. Volume data were transformed from native space into MNI space using a nonlinear volume-based registration. For accurate cross-subject registration of cortical surfaces, a multimodal surface matching (MSM) algorithm (Robinson et al., 2014) was used. The

Analysis of fMRI data:
Functional images were minimally preprocessed using the HCP pipeline (Glasser et al., 2013).
Preprocessing included correction for spatial distortions due to gradient nonlinearity and b0 field inhomogeneity, fieldmap-based unwarping of EPI images, motion correction, brain-boundary-based registration of EPI to structural T1w scans, non-linear registration to MNI space, and grand-mean intensity normalization. Data from the cortical gray matter ribbon were projected onto the surface and then onto the standard grayordinates space. Data were minimally smoothed by a 2mm FWHM Gaussian kernel in the grayordinates space. Thus, smoothing was constrained to the cortical surface mesh in each hemisphere. Data were cleaned up for artifacts and structured noise using sICA+FIX.
The preprocessed functional time-series were entered into a general linear model (GLM) to estimate functional activities in each vertex/voxel in each run (Barch et al., 2013). Two regressors/predictors were included in the GLM design of the language task -Story and Math. Each predictor covered the duration of a block (~ 30 s). Two regressors/predictors were included in the GLM design of the social task -Social and Random. Each predictor covered the duration of a single video clip (20 s). All regressors were convolved with a canonical hemodynamic response function and its temporal derivatives. The time-series were temporally filtered with a Gaussian-weighted linear highpass filter with a cutoff of 200 s, to remove low-frequency drifts/fluctuations presumably unrelated to the task design. The time-series were also prewhitened to remove temporal autocorrelations in the fMRI signal. For three comparisons ('story vs. baseline', 'story vs. math', and 'social vs. random'), the contrast of parameter estimate (COPE) was computed based on beta values of the GLM. The language task did not have typical fixation blocks (there was only an 8-second 'get ready' countdown at the beginning of the run), and the baseline condition in this task represented the mean activity across all time-points in each run. Fixed-effects analyses were conducted to estimate the average effects across runs within each subject, then mixedeffects analyses treating subjects as random effects were conducted to obtain group-average maps. The group-average maps were Cohen's d effect size maps. Cohen's d in each vertex/voxel was computed as the mean COPE (across subjects) divided by the standard deviation.

Data/Code availability:
All data used in this manuscript are part of publicly available and anonymized HCP database (https://www.humanconnectome.org). All analysis codes are available for sharing upon request. Two language tests were used for assessing the linguistic abilities of subjects. In the vocabulary comprehension test, subjects were presented with an audio recording of a word and they had to select the picture that best matched the meaning of the word. In the reading recognition test, subjects were asked to read and pronounce words as accurately as possible. For each subject, the age-adjusted scores in the two tests were z-normalized and averaged. Based on average scores, we selected two groups of subjects who either had high score (one SD above mean, n = 152) or low score (one SD below mean, n = 174). Similarly, two groups of subjects were selected based on accuracy in the social task (high-accuracy group: n = 430, low-accuracy group: n = 259). Since accuracy did not have a normal distribution (p < 0.05, Kolmogorov-Smirnov normality test), we used median ± median absolute deviation as cutoff points. The bar plots in panels a-b demonstrate the functional asymmetry values for two groups of subjects in eight language areas. In each area, the statistical comparison was based on two-sample t-test The language network largely overlapped with our functionally defined language areas (green borders).

References
(b) Responses of LH ROIs in the 'story vs. baseline' and 'math vs. baseline' contrasts of the language task.
Error bars indicate one standard error of the mean across subjects. In p47r and IFSa, responses were significantly higher in the math condition compared to the story condition (FDR-adjusted p < 0.0005; paired t-test). In other ROIs, responses were significantly higher in the story condition compared to the math condition (FDR-adjusted p < 0.0005; paired t-test).