Pupillometry reveals effects of pitch manipulations both within and across words on listening effort and short-term memory

doi:10.21203/rs.3.rs-3249989/v1

Download PDF

Article

Pupillometry reveals effects of pitch manipulations both within and across words on listening effort and short-term memory

https://doi.org/10.21203/rs.3.rs-3249989/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 30 Sep, 2024

Read the published version in Scientific Reports →

You are reading this latest preprint version

For individuals with hearing loss, even successful speech communication comes at a cost. Cochlear implants transmit degraded acoustic, specifically pitch, information, which demands extra and sustained listening effort. The current study hypothesized that abnormal pitch patterns contribute to the additional listening effort, even in non-tonal language native speaking NH listeners. We manipulated the fundamental frequency within-and-across-words, while participants listen and repeat (simple intelligibility task), or listen, repeat, and later recall (concurrent encoding task) the words. In both experiments, the F0 manipulations resulted in small changes in intelligibility but no difference in free recall or subjective effort ratings. Pupillary metrics were more sensitive to these manipulations: peak pupil dilations were larger when words were monotonized (flat pitch contour) or inverted (the natural contour flipped upside-down), and larger when successive words were organized into a melodic pattern. Therefore, natural or expected pitch contour of a word can participate to its identity and facilitate its matching and retrieval from the phonological representation stored in the long-term memory. Consequently, degrading words’ pitch contour can result in extra listening effort. Our results call for solutions to improve pitch saliency and naturalness in future development of cochlear implants signal processing strategies, even for non-tonal languages.

Biological sciences/Neuroscience/Cognitive neuroscience/Perception

Biological sciences/Psychology/Human behaviour

Hearing loss is recognized by the WHO as one of the top 5 leading causes of disability, and its effects will only grow as we live in an increasingly noisy world ¹. On top of the practical challenges of living with reduced hearing, hearing loss is often related to negative cognitive and psychosocial impact. For instance, hearing loss has been identified as the leading modifiable risk factor for dementia ^2,3. Though many challenges remain in addressing this issue, medical interventions do exist, such as hearing aids and, in cases of severe to profound sensorineural hearing loss, cochlear implants (CIs).

CI recipients experience good speech-in-quiet performance. However, individual differences in outcomes are substantial, both in speech performance and life quality ^4–7. Specifically, most CIs on the market follow envelope-based coding strategies, and eliminate the fine structure necessary to extract a precise fundamental frequency (F0) ^8–10. The result is that, despite generally accurate timing cues, the quality of pitch encoding is severely degraded compared to that of normally hearing (NH) listeners. This significantly limits CI users’ abilities in domains that rely on a fine sense of pitch, such as music, speech prosody and emotion, speaker identity, and segregating competing speakers in a conversation ^11–15. This problem is not unique to CI users: hearing impaired (HI) listeners have decreased temporal fine structure processing and broadened peripheral filters ^16,17. For young children, a fine representation of voice pitch information has shown to be primordial, as the exaggerated prosody of infant-directed speech is thought to play a key role in language acquisition ^18–21. Therefore, pitch is an important perceptual feature that is perturbed in multiple ways and for multiple kinds of hearing loss, with deleterious effects on early language learning as well as cognitive aging.

For both individuals with hearing loss and normally hearing listeners in non-optimal situations (for instance low-quality phone calls), degraded pitch information may not only affect hearing performance, but could also contribute to elevated effort and fatigue, even when speech recognition scores are seemingly unaffected ^22–24. The relationship between signal quality and listening effort can be understood using the ease of language understanding (ELU) model: when the speech signals are clean, perceived signals can be matched to long-term phonological representations automatically, without engaging explicit cognitive resources; however, when speech is distorted, for instance when the acoustic signal is degraded by masking noise or spectral smearing, a mismatch will occur between the perceived signal and the template stored in the long-term memory. Resolving this mismatch requires extra processing and more cognitive resources, leading to more effortful listening ^25–27. Subsequently, this additional demand might leave fewer resources for other secondary tasks, such as recall and information synthesis.

The behavioral cost of elevated listening effort has been demonstrated and quantified in past studies using dual-task paradigms. For instance, studies using the sentence-final word identification and recall (SWIR) test have shown that recall can be impaired in difficult listening conditions, even though intelligibility scores remained unchanged ^28–30. Therefore, it is likely that using a similar dual-task paradigm, degraded pitch information would increase the listening effort during speech decoding, leaving fewer resources for memorizing the items. In addition, there is some evidence that certain pitch patterns in spoken material (across words) can improve immediate free recall for NH listeners ^31–35. It is unclear whether such phenomena would hold with abnormal pitch patterns (within or across words) but this represents an additional route by which pitch degradations could impair the secondary task in a dual-task paradigm.

Elevated listening effort has a physiological ‘footprint.’ Pupillometry has been shown to be a valid correlate of listening effort in different listening conditions ^24,36–40. Event-evoked pupillary responses typically increase with increased task demands, for instance low signal-to-noise ratio (SNR), reverberation, spectral degradation, etc, until an ‘inversion point’ where the participants ‘give up’ ^41,42. Therefore, event-evoked responses like peak pupil dilation (PPD), mean pupil dilation (MPD) and peak latency are interpreted as indications of effort. Recently, more studies have started to investigate pupil responses during concurrent tasks, in an effort to understand pupil dynamics in ecological conditions ^43–46, because rarely in real life are people challenged by one task at a time. For instance, ⁴⁶ found that the task complexity (listening, repetition, or learning) and the acoustic adversity (noise level) affected different elements of the pupil traces. While more difficult background noise was associated with higher PPD, accumulating memory load was associated with increasing baseline pupil response (plateauing in the second half of a list). Therefore, pupillometry is sensitive to different types of cognitive load. In the current study, we replaced SNR (noise) manipulations with F0 manipulations, at both within-word or across-word levels to address the hypothesized impact of pitch distortion on listening effort and memory performance in a NH sample.

The first experiment focused on within-word pitch information by manipulating the F0 contour of each word presented. We hypothesized that monotonized words would be less accurately recalled, and lead a greater pupil dilation, than naturally intonated words, because participants would need to engage explicit cognitive resources to resolve the mismatch between the somewhat artificial (robotic-like) voice quality of the words and their stored phonological representations. We also included words with an inverted F0 contour, which, despite having the same pitch range and formant changes as the naturally intonated words, violate the stored template for the words. Both monotonized and inverted versions should pose a conflict with the stored representation of these words, just like speech degraded by noise or spoken in accents ^36,37,47. On the contrary, exaggerated pitch contours may be processed with more ease than normally-intonated words, leading to better memory and smaller pupil dilation. While more speculative, we made this prediction based on the exaggerated intonation present in ‘motherese speech’, i.e. speech directed to infants and toddlers ⁴⁸ as well as caricatures ^49,50 where exaggerated vocal traits may help (not hinder) recognition despite the acoustic distortions they produce.

The second experiment focused on across-word (“suprasegmental”) pitch sequences by manipulating mean F0 of successive words across the list of ten. We compared a fixed condition (identical to the monotonized condition of experiment 1) with a melodic condition, in which monotonized words were presented at different mean F0s (“notes”), making an arpeggio-like sequence of pitches. It was not known how this pitch pattern would affect recall, if at all: there are various studies showing pitch patterns to be either helpful, irrelevant, or even detrimental to short-term memory of words ^{31,32,34,35,51–53}.

These manipulations will help us understand whether access to salient pitch information supports word decoding and storage. Results will guide us to improve front-end speech processing strategies to improve acoustic information transfer for assistive hearing devices such as hearing aids and CIs.

Experiment 1: Within-word pitch manipulations

Behavioral data: intelligibility

The results are summarized in Table 1 and Fig. 1 (top left). As hypothesized, there was a main effect of pitch manipulation [χ²(3) = 13.9, p = 0.003], caused primarily by the exaggerated pitch condition leading to better intelligibility than the monotone and inverted conditions (p < 0.010). Bear in mind, however, that the words (being presented in quiet) were very well recognized (close to ceiling), so this difference was close to negligible, with the difference amounting to about 2% (intelligibility was > 95% overall). The task (i.e. whether listeners simply repeated words or repeated while memorizing them) had no effect, but there was a main effect of position [χ²(1) = 12.6, p < 0.001], reflecting that intelligibility varied somewhat throughout a list (bottom-left). We did not grant much credit to this position effect as a linear regression did not reach significance; the effect could be driven by a few occasional mistakes occurring earlier in the list. Finally, intelligibility did not correlate with the subject’s age [p = 0.687] (Supplementary material 2).

Table 1

Results of the statistical analyses of behavioral and pupillary data in both experiments. Significant effects (p < 0.05) are marked with an asterick *.
Behavioral data
	Exp1: within-word F0 manipulation			Exp2: across-word F0 manipulation
Effect of adding fixed term to the model	intelligibility	recall	subjective	intelligibility	recall	subjective
recall	χ²(1) = 0.7, p = 0.416		χ²(1) = 1111.9, p < 0.001*	χ²(1) = 2.4, p = 0.122		χ²(1) = 367.3, p < 0.001*
pitch	χ²(3) = 13.9, p = 0.003*	χ²(3) = 0.3, p = 0.954	χ²(3) = 0.5, p = 0.909	χ²(1) = 3.2, p = 0.074	χ²(1) = 1.5, p = 0.219	χ²(1) = 1.2, p = 0.271
position	χ²(1) = 12.6, p < 0.001*	χ²(2) = 259.7, p < 0.001*		χ²(1) = 1.4, p = 0.239	χ²(2) = 114.3, p < 0.001*
recall x pitch	χ²(3) = 2.2, p = 0.528		χ²(3) = 0.2, p = 0.971	χ²(1) = 3.5, p = 0.061		χ²(1) < 0.1, p = 0.811
recall x position	χ²(1) = 0.7, p = 0.395			χ²(1) < 0.1, p = 0.999
pitch x position	χ²(3) = 6.3, p = 0.098	χ²(6) = 5.0, p = 0.547		χ²(1) = 3.7, p = 0.055	χ²(2) = 5.7, p = 0.059
3-way	χ²(3) = 1.3, p = 0.738			χ²(1) < 0.1, p = 0.810
Pupillary data
	Exp1: within-word F0 manipulation			Exp2: across-word F0 manipulation
Effect of adding fixed term to the model	baseline	PPD	peak latency	baseline	PPD	peak latency
recall	χ²(1) = 917.3, p < 0.001*	χ²(1) = 1.0, p = 0.319	χ²(1) = 11.3, p < 0.001*	χ²(1) = 178.4, p < 0.001*	χ²(1) = 8.6, p = 0.003*	χ²(1) = 3.1, p = 0.080
pitch	χ²(3) = 2.0, p = 0.577	χ²(3) = 12.2, p = 0.007*	χ²(3) = 5.4, p = 0.148	χ²(1) = 1.8, p = 0.181	χ²(1) = 6.8, p = 0.009*	χ²(1) = 0.5, p = 0.489
position	χ²(1) = 2.5, p = 0.114	χ²(1) = 186.8, p < 0.001*	χ²(1) = 68.7, p < 0.001*	χ²(1) = 0.2, p = 0.634	χ²(1) = 34.6, p < 0.001*	χ²(1) = 12.8, p < 0.001*
recall x pitch	χ²(3) = 8.5, p = 0.037*	χ²(3) = 2.1, p = 0.543	χ²(3) = 10.3, p = 0.016*	χ²(1) = 1.2, p = 0.267	χ²(1) < 0.1, p = 0.961	χ²(1) < 0.1, p = 0.818
recall x position	χ²(1) = 192.1, p < 0.001*	χ²(1) = 15.9, p < 0.001*	χ²(1) = 0.8, p = 0.384	χ²(1) = 55.0, p < 0.001*	χ²(1) = 1.7, p = 0.191	χ²(1) = 0.1, p = 0.7
pitch x position	χ²(3) = 3.3, p = 0.35	χ²(3) = 1.5, p = 0.681	χ²(3) = 9.2, p = 0.027*	χ²(1) = 0.2, p = 0.656	χ²(1) = 1.1, p = 0.288	χ²(1) = 1.7, p = 0.192
3-way	χ²(3) = 2.1, p = 0.556	χ²(3) = 1.1, p = 0.768	χ²(3) = 6.3, p = 0.098	χ²(1) < 0.1, p = 0.972	χ²(1) < 0.1, p = 0.772	χ²(1) = 0.5, p = 0.462

Behavioral data: recall

The recall data demonstrated the expected primacy/recency effect (bottom-middle, Figure.1), which was revealed as a quadratic main effect of position [χ²(2) = 259.7, p < 0.001]. But the pitch manipulation had no effect, nor did it interact with position. Participants recalled about 6.5 words per list (top-middle, Figure.2). Average overall recall decreased with the subject’s age [r2 = 0.17, p = 0.015] (Supplementary material 2).

Behavioral data: subjective rating

As expected, participants found the recall task more effortful than the repeat-only task [χ²(1) = 1111.9, p < 0.001] (top-right, Figure.1), but pitch manipulation had no role. Average effort was unrelated to the subject’s age [p = 0.427] (Supplementary material 2).

To summarize the behavioral portion of this experiment, except for slight changes in intelligibility (questionable due to ceiling effects), within-word pitch manipulations had no impact on recall and were equally effortful. All other effects, e.g. recall exhibiting a typical U-shape, recall worsening with age, and recall being more effortful than the repeat-only task, were expected from the literature using the same paradigm ⁴⁶.

Pupillary data

The pupil traces are shown in Figs. 2 and 3, aligned at the response onset, i.e. when subjects were instructed to repeat the word back to the experimenter. On this scale, the word presentation ended at -1 second, and it is clear that the pupil kept on dilating for another half a second after that (bottom panels). It is also apparent that the pupil traces were on average higher in the recall task than in the repeat-only task (top panels). Critically, it is not because participants happened to have different baseline to begin with: as illustrated in the most-top-left panel of Fig. 3, the pupil traces started from a similar range at the beginning of a list, but progressively departed from each other depending on the task. The pupil progressively relaxed throughout the list when subjects only repeated the words, while dilation was maintained at a high level (and progressively incremented) when subjects had to keep rehearsing the words in their mind in addition to repeating them. This also led to PPDs that quickly dwindled in amplitude, resulting in a different pattern as a function of position than in the repeat-only task.

To provide more quantitative assessments of these observations, three metrics were focused on: baseline, PPD amplitude, and PPD latency.

Pupillary data: baseline

Regarding the baseline of pupillary responses, the full results are displayed in Table 1. There was a main effect of task [χ²(1) = 917.3, p < 0.001] confirming that the pupil diameter was overall higher in the recall task than in the repeat-only task (top-left, Figure.4). This was because the pupil kept on increasing slowly throughout the list in the recall task, whereas it progressively relaxed in the repeat-only task (top-right, Figure.4), resulting in an interaction between task and position [χ²(1) = 192.1, p < 0.001].

There was no main effect of the pitch manipulation, but it interacted modestly with the task [χ²(3) = 8.5, p = 0.037]. The effect of recall was more prominent with the mono than with the inve condition (top-left, Figure.4).

Finally, the average baseline decreased for older subjects [r2 = 0.23, p = 0.004] (by 2 mm across a 35-year range - see Supplementary material 2).

Pupillary data: PPD

For results regarding PPD of pupillary responses, there was a main effect of position [χ²(1) = 186.8, p < 0.001], confirming that PPD dropped in amplitude very quickly from the first to the second word, and kept on decreasing throughout the rest of the list. This effect was dependent on the task [χ²(1) = 15.9, p < 0.001], being exacerbated in the recall task (middle-right, Figure.4).

One of the key results was that, despite there being no difference in behavioral recall for the different pitch conditions, within-word pitch manipulation did impact the size of the PPD [χ²(3) = 12.2, p = 0.007]. Post-hoc comparisons revealed that PPD was smaller for the into than the exag condition, itself smaller than both inve and mono conditions (which did not differ from each other). In other words, if we take the size of PPD as a metric of the effort engaged in decoding a given word, we would conclude that any degradation from the original pitch contours induced additional effort, and even more when these degradations were inconsistent with the directions of the original contour (middle-left, Figure.4).

Averaged PPD amplitude did not correlate with subjects’ age [p = 0.306] (Supplementary material 2).

Pupillary data: peak latency

A main effect of task [χ²(1) = 11.3, p < 0.001] suggested later pupillary peak latency in recall than in repeat-only (bottom-left, Figure.4), but this occurred only for the two conditions that led to the largest PPDs, namely inve and mono conditions, resulting in an interaction with pitch [χ²(3) = 10.3, p = 0.016].

Position also led to a main effect [χ²(1) = 68.7, p < 0.001], reflecting earlier and earlier peaks from the beginning to the end of a list (bottom-right, Figure.4), and this effect depended somewhat on the pitch manipulation [χ²(3) = 9.2, p = 0.027] (not shown). This latter interaction appeared quite complex and to a small degree also dependent on the task as well (the 3-way interaction approaching significance, p = 0.098). For simplicity (and also because this interaction with position did not occur in Exp2), we did not explore it further.

Averaged PPD latency did not correlate with subjects’ age [p = 0.548] (Supplementary material 2).

Experiment 2: Across-word pitch manipulations

Behavioral data: intelligibility

No significant main effect or interactions were found (Table 1). There was a trend such that the melodic condition was slightly more intelligible than the fixed condition (p = 0.074) but the effect size was small: it was only about 1% and intelligibility was > 95% overall, as in experiment 1 (top-left, Figure.5). Similarly, pitch tended to interact with task (p = 0.061) and with position (p = 0.055) in this narrow and close-to-ceiling range. Average intelligibility did not depend on the subject’s age [p = 0.638] (Supplementary material 2).

Behavioral data: recall

The recall data demonstrated primacy and recency effects (bottom-middle, Figure.5), yielding a main effect of quadratic position (Table 1). The pitch manipulation had no overall effect, and its interaction with position did not reach significance (p = 0.059). Subjects recalled about 6.5 words per list (top-middle, Figure.6), and this performance decreased with age [R² = 0.16, p = 0.045] (Supplementary material 2).

Behavioral data: subjective rating

As in experiment 1, all subjects found the recall task more effortful than the repeat-only task [Chi2(1) = 367.3, p < 0.001] (top-right, Figure.5), but pitch manipulation had no effect (Table 1). Average effort was not related to the participant’s age [p = 0.281] (Supplementary material 2).

To summarize, except for slight changes in intelligibility (which never reached significance), the melodic manipulation had no impact. All other effects, e.g. recall exhibiting a typical U-shape, recall worsening with age, and recall being more effortful, replicated previous results ⁴⁶.

Pupillary data

The averaged pupil traces are shown in Figs. 6 and 7. Similarly, the pupil diameter increased during the presentation of a word and kept on dilating for another half a second after that. This pattern is not so obvious when looking at the raw measurements in mm (top panels) but quite clear in the baseline-corrected traces (bottom panels). The raw measurements, on the other hand, illustrate clearly that the pupil was more dilated in the recall task than in the repeat-only task. Once again, this was not a spurious baseline difference between conditions: as illustrated in the most-top-left panel of Fig. 7, participants started roughly at the same baseline, but the pupil traces progressively departed from one another, depending on the task.

To provide quantitative assessments of these observations, three metrics were extracted.

Pupillary data: baseline

The full results are displayed in Table 1. There was a main effect of task [χ²(1) = 178.4, p < 0.001] confirming that the pupil diameter was about 0.2 mm higher in the recall than in the repeat-only task (top-left, Figure.8). As described earlier, this was because the pupil kept on increasing slowly throughout the list in the recall task, whereas it progressively relaxed in the repeat-only task (top-right, Figure.8), resulting in an interaction between task and position [χ²(1) = 55.0, p < 0.001].

There was no effect of pitch manipulation or any interaction. The average baseline decreased for older subjects [r2 = 0.22, p = 0.020] (by about 2 mm across 30 years - Supplementary material 2).

Pupillary data: PPD

There was a main effect of position [χ²(1) = 34.6, p < 0.001] confirming that PPD diminished in amplitude very quickly from the first to the second position, but was relatively stable after that (middle-right, Fig. 3). In contrast to Exp.1, the effect of task was more homogeneous across the list, and so it appeared as a main effect (rather than an interaction with position) [χ²(1) = 8.6, p = 0.003]. PPD was on average 0.02 mm larger in the recall than in the repeat-only task. Note that this is 10 times smaller than its effect on the baseline.

More to the heart of this study, the pitch manipulation impacted PPD amplitude [χ²(1) = 6.8, p = 0.009]. The PPD was elevated in the melodic relative to the fixed condition (middle-left, Fig. 3). Also, averaged PPD amplitude tended to decrease with older subjects [r2 = 0.17, p = 0.043] (Supplementary material 2).

Pupillary data: peak latency

The main effect of task did not reach significance [χ²(1) = 3.1, p = 0.080] but would have suggested the same direction as in Exp.1: about 50-ms later peaks in recall than in repeat-only (bottom-left, Fig. 8). Position also led to a main effect [χ²(1) = 12.8, p < 0.001], reflecting earlier and earlier peaks from the beginning to the end of a list (bottom-right, Fig. 8).

More importantly, pitch did not result in a main effect and did not interact with task or position. Average PPD latency did not correlate with subjects’ age [p = 0.105] (Supplementary material 2).

In Experiment 1, pupil traces differed markedly depending on the task (recall vs repeat), consistent with past studies that investigate the effect of recall on pupillary response. Although baseline dilation increased at the beginning of each list for both tasks, they diverged after the first 2–3 words. The repeat-only task was extremely easy, and as a consequence the pupil relaxed, ending up 0.2 mm lower than where it started. In contrast, recalling as many words as possible demanded sustained cognitive resources, leading to a continued increase in pupil diameter. Initial PPD values were high, but these decreased over the course of the 10-word list, meaning that average PPD could be inordinately affected by the first word. These effects of task and position are interesting and generally warn us about the limitations of a single metric such as PPD ^43,46,54. In ecological situations, one is often faced with multi-tasking, and measuring how the averaged diameter varies over long periods of time can yield useful information.

More to the point of this study, we found that pupillometry was sensitive to subtle effects generated by the F0 contour within words. Consistent with our hypothesis for pupillometry outcomes, the PPD was larger in both monotonized and inverted conditions, suggesting that these conditions required extra resources to process. For behavioral outcomes, we replicated the poorer intelligibility of these two conditions as it has been found in sentences with noise ^55,56, though the impairment was minimal in our current study because individual words in quiet can hardly be misunderstood (ceiling performance). Note that this was by design: had we presented the words in background noise (to lower intelligibility), the periodicity information would have been less well extracted (or would have served other purposes, e.g. segregation).

Additionally, we observed two interesting interactions between pitch and task conditions: one for the baseline and the other for PPD latency. The latency effect is relatively easy to interpret since the effect of recall (delaying the peak dilation) was seen exclusively for the conditions that were already difficult to process, namely monotonized and inverted pitch contours. The effect of the task would therefore exacerbate any initial difficulty in word decoding. The baseline effect is not as easy to interpret: it is possible that the absence of pitch information (or more accurately the fact that it is uninformative) in monotonized words would make them even more difficult to decode than inverted words (which have some - but potentially misleading - pitch information). In turn, expectation to the extra difficulty in the whole monotonized condition would exacerbate the overall effort induced by recall ⁵⁷. In other words, both interactions might point to the idea that the recall task is more costly for listening situations that were already taxing resources.

These results demonstrate that, even in normally hearing non-tonal language native speaking individuals, pupillometry is sensitive to unnatural and distorted F0 inflections, and we interpret differences in response to reflect explicit investment of cognitive resources. This would be explained by the ELU model in the following manner: the phonological representation of monotonized or inverted words becomes harder to match with a “template” of a known word in the listener’s long-term memory. To make the match, explicit engagement of extra cognitive resources is necessary, and this extra engagement is shown in the task-evoked pupillary response (even when behavioral response and subjective reflection do not capture it).

Thus, abnormal pupil responses to words in quiet may indicate that performance will worsen in noisy situations, even though in our experiment there was little effect of pitch conditions on intelligibility, memory, or subjective difficulty rating. Most likely, changes in behavioral and subjective responses can in principle occur without adding noise, but they might require more power and adjustments to the experimental design such as a greater sample size or a longer wait time before testing recall.

In Experiment 2, many findings from Experiment 1 were replicated: the effects of task and position were identical for the baseline and very similar for the PPD amplitude (but affecting the entire list rather than changing the slope across word positions).

More to the point of this study, we found again that pupillometry was sensitive to pitch manipulations, this time across words instead of within words. Given the “across-word” nature of the manipulation, this is a different interpretation, no longer about the phonological representation of words but more about their encoding as a sequence forming a melody via their respective F0s. A complete understanding of this phenomenon is not trivial.

Although these manipulations did not have behavioral consequences here, we know that they can in principle affect memory performance. Sares et al. ³⁵ tested free recall performance after presenting word lists with different pitch sequences, and found that there was an improvement in free recall only when the sequences indicated a grouping (arpeggios). Since the melodic condition in this experiment had a similar pattern to the arpeggios in Sares et al. (2023), one might have expected the melodic condition to lead to better recall than the fixed condition. Not only was this not the case, but performance in the melodic condition tended to be worse. One important parameter differentiating the two designs is speed of presentation: while in Sares et al. (2023) the words were presented close in time (less than 1 second apart), in a way which could facilitate grouping and pitch pattern recognition, here the words were presented around 5 seconds apart (1s waitpeak + participants repeating back the words), a timescale which may be perceived quite differently ⁵⁸. The flattened profile of ten words in a sequence with short intervals makes it difficult to retain them in memory since items that are more distinct from one another could be easier to store and retrieve than items that share common features. For instance, several studies showed that, in NH listeners, the recall of words that rhyme is poorer than the recall of words that do not rhyme (Baddeley, 1966; Conrad & Hull, 1964; Nittrouer et al., 2013; Salamé & Baddeley, 1986). Another possibility could be that on this larger timescale, the role of pitch is not to facilitate memory encoding, but rather to support participant’s engagement with the task. Previous study has shown that pupillary response is sensitive to changes in task engagement even when behavioral performance is the same (similar in our case) ⁵⁷. However, we did not observe differences in baseline pupil diameter between two manipulations that were typically related with anticipating or mobilizing attention ^63–65. We did not have any other markers to estimate task engagement, nor a physiological correlation of arousal and stress (e.g. salivary cortisol, skin conductance, or heart rate recordings). The melodic condition resulted in larger PPDs but it remains unclear whether this is a sign of additional effort in processing and storing words, or whether it reflects a more enthusiastic engagement towards this condition.

In summary, pitch manipulations applied in both Experiment 1 and Experiment 2 (within-word and across-word) had small effects on intelligibility (in ceiling region) and did not significantly affect immediate recall or subjective difficulty ratings. It is surprising that immediate recall did not show any significant difference, especially considering how similar dual-task behavioral paradigms have shown to be sensitive to different acoustic manipulations ^28,29. This lack of sensitivity could be due to the use of words rather than sentences. In our case, the recall was likely tapping into a phonological loop of recently stored monosyllabic words, so the impact of acoustic manipulation could be more heavily influenced by the recency effect. When using longer and more complex stimuli, for instance the SWIR, the impact of an experimental manipulation might rely more on the primacy effect, where sentence-final words are transferred to long term memory. Additionally, the effect size of our acoustic manipulations might be smaller compared to other manipulations (i.e., SNR, noise reduction turned on and off), hence harder to observe with current statistical power.

On the other hand, the pupil responses did differ, suggesting that pupillary responses might be sensitive to subtleties in the allocation of cognitive resources. These subtleties are meaningful within the ELU framework: the more matched the acoustic inputs with the stored template of pitch contour, the less need for explicit cognitive resources to resolve the mismatch, hence the smaller the pupillary response. This is presumably why the inverted and monotone pitch contours led to increased pupil dilation and response latency, especially during the recall task, as well as slightly decreased intelligibility– we could perhaps call this “unproductive effort.” Experiment 2 showed increased pupil dilation and response latency for the melodic pitch condition, and intelligibility was slightly increased (though not significantly). This is the opposite relationship to experiment 1; and could be thought of as “productive effort.”

The dichotomy between productive and unproductive effort is seen elsewhere in past studies using pupillometry to quantify effort. For instance, in speech recognition tasks, elevated noise during listening can lead to greater pupil dilation and poorer intelligibility or recall, up to a point ^37,42. In memory tasks, greater pupil dilation is seen for words that are recalled compared to those that are not recalled (see details of the replication results in Supplementary material 1) and also in previous work ⁵⁴. Thus, though pupillometry is a powerful and sensitive technique to register fluctuation in physiological responses to cognitive demands, it is difficult to interpret pupil signals in the absence of the task demands and behavioral outcomes. An increase in task-evoked pupillary response or cognitive resources expenditure is not necessarily a negative marker; it can reflect engagement and be followed by successful completion of a more complex task. Ultimately, it is individual differences in cognitive and motivational status that decide whether an event is perceived as negative or positive by the listeners (Carolan et al., 2022; Herrmann & Johnsrude, 2020; Pichora-Fuller et al., 2016).

The impact of F0-related acoustic manipulation on intelligibility and pupillary response found in this paper have not been reported systematically in previous studies. While the results are consistent with ELU framework in general, there has not been enough research conducted on the contribution of F0 to the internal identity of words. For non-tonal language speakers, F0 contour has not been considered as important as phonemic components (i.e., vowels and consonants), therefore its encoding has not been investigated as thoroughly as it has been in tonal languages. Without knowing the encoded ‘template’ of an F0 contour for a given word, it is difficult to predict the impact of acoustic degradation. Our results seem to suggest that even though F0 contour does not play a phonemic role in non-tonal language phonology, distorting the contour still has an impact on single word and word list recognition and listening effort. In Exp1, inverting and monotonizing the F0 contour introduced mismatches that require extra cognitive resources to resolve (as shown by pupillometry). Exaggerating the contour, however, did not even though it reflected some deviation from the typical contour. This is a promising result, which is in line with the correct performance obtained with caricatures ^49,50 and reinforces the notion that pitch exaggerations should be further explored as a solution for CI difficulties in this domain (e.g. He et al. ⁶⁸for a similar endeavor in tonal languages). The effect seen in Exp2 for the melodic condition, on the other hand, would be difficult to implement ecologically and needs to be understood as a complex interaction among attention, engagement, arousal, anxiety, and effort ²⁶. Without other markers of engagement and attention, it is difficult to pin down whether the benefit of adding a melodic pattern over a series of words is due to increasing attention or providing some pitch variability similar to natural speech. We doubt the latter: overlaying a pattern of F0 variation on isolated words is not the same as fully restoring a natural pitch contour to a connected utterance, because natural sentences utilize pitch contour to transmit extra information (i.e., prosody, emotion and intonation). It could be expected, based on previous studies, that restoring natural contour will alleviate more listening effort, if not improving intelligibility significantly, even in non-tonal language ⁶⁹. But Exp2 suggests that introducing some pitch variability affects the processing of a series of speech inputs, even if that pitch variability doesn’t resemble natural prosody.

Note that the current experiment was conducted in NH listeners. Future studies should investigate whether these findings extend to hearing impaired populations and specifically CI users. The auditory inputs from a CI contain less salient and sometimes distorted F0 cues (e.g. incomplete array insertion). Although many factors contribute to the challenging and effortful speech recognition in CI users, the importance of F0 saliency and fidelity cannot be ignored in speech recognition and associated listening effort. This is not only due to the importance of F0 in transmitting prosodic information (i.e., intonation, emotion, etc.), but also in decoding the words themselves, since a word’s F0 contour is part of its identity. Whether CI users completely ignore pitch in their phonological representation, or whether they struggle with it because it lacks discriminative power, is an open question which we hope to address in a future study.

Participants

In Experiment 1, we recruited a group of 35 adults (12 male, 23 female) between the ages of 18 and 51 (mean ± SD = 25.7 ± 8.2). Participants had normal hearing (NH); defined as having no history of audiological problems and having binaural pure tone thresholds better than 30 dB HL at 0.25, 0.5, 1, 2, 4, 8 kHz. All participants were native speakers of either French or English (the two most common languages in the city; 14 native French speakers). The experiment was always run in their native language (another study is in preparation to investigate language background effects in such paradigms).

In Experiment 2, we recruited 25 adults (10 male, 15 female) between the ages of 18 and 51 with (mean ± SD = 24.6 ± 7.4) years. All but three had participated in experiment 1, but the two experiments were treated separately. No statistical comparison was made between the two experiments.

This work received ethical approvals from McGill University Faculty of Medicine Research Ethics Board (IRB) under the number A05-B11-18B. Both studies were performed in accordance with McGill University ethical guidelines and regulations. Prior to the experiment, participants were given enough time to read the information sheet and consent form approved by the ethics board. All gave written informed consent for their participation.

Stimuli

Word stimuli were consonant-nucleus-consonant (CNC) words ⁷⁰ recorded by a male native American English speaker, and Fournier words ⁷¹ recorded by a male French speaker. The STRAIGHT algorithm ⁷² was then used to manipulate the F0 of each word.

In experiment 1, there were 4 conditions: 1) a monotonized condition (mono) where F0 remained steady at a fixed value for all words (this value was 121 Hz as the mean F0 of the entire English material, or 101 Hz as the mean F0 of the entire French material); 2) a naturally intonated (into) condition where F0 was unprocessed; 3) an exaggerated (exag) condition where the fluctuations of the naturally intonated F0 pattern relative to the monotone were doubled on a logarithmic scale (since pitch is perceived logarithmically); 4) an inverted condition (inve) where the fluctuations of the naturally intonated F0 pattern were flipped upside-down around the mean F0 for that word. These corresponded to the parameter k being 0, 1, 2, and − 1, respectively for monotonized, intonated, exaggerated, and inverted, in the Eq. (1) below:

\(F0’ = ref \times 2^(k \times log2(F0/F0m\left)\right)\) Eq. (1)

where F0’ is the manipulated F0 contour, ref the mean F0 of 121 or 101 Hz depending on the language material (due to different average F0 in English and French words), k the parameter differentiating the 4 conditions, F0m the mean F0 respective to each sentence, and F0 the original F0 contour.

Note that all these manipulations have been validated before, in the context of speech intelligibility for NH adults ⁵⁵. Finally, words were grouped into lists of ten.

In experiment 2, all words were first monotonized to a fixed value (ref). Then the F0 of these words was manipulated in 2 conditions: a fixed condition where F0 remains steady for all words; and a melodic condition where F0 of each word within the 10-word block follows the notes: do, mi, sol, re, fa, la, re#, fa#, si and sol#, where do was based on the ref value.

Procedure

After Participants sat on a stable chair in a soundproof room, 2.5m in front of a 35-inch screen monitor and wearing an infrared binocular eye tracker (Tobii Glasses Pro2, 50 Hz sampling rate). The room and screen luminance levels were adjusted to reach 80 lx (measured using a luxometer with the sensor positioned at the same height of the participants’ left eye and facing the screen). The luminance levels were fixed throughout the experiments, to avoid changes in light level inducing task-unrelated pupillary response and to maximize task-related responses ⁷³. All audio stimuli were presented through a Beyer Dynamics DT 990 Pro headphone via an external soundcard (Edirol UA), calibrated at 65 dB SPL. Experiments were run in Matlab 2016b, using Psychtoolbox and custom scripts.

On each block, before listening, a word on the screen indicated that the participant should try to remember the words (RECALL, printed in red) or not (NO RECALL, printed in black). After the notification, there was a 3 second pause, and then participants listened to a list of 10 words. Participants were instructed to fixate on a light gray fixation cross displayed at the center of the dark gray screen during this time. 1.5 seconds before the onset of each word presentation served as time for pupil to restore from the previous trial (0.5s) and baseline measurement (1s). At the offset of the word presentation, participants were instructed to not respond immediately but keep fixating on the gray fixation cross for 1s to wait for the late-occuring pupil peak to appear (‘waitpeak’ period). At the end of the waitpeak stage, participants were prompted by a light gray circle at the center of the screen to repeat the word aloud. The spoken responses were scored by the experimenter based on whether the word was correctly repeated. In blocks requiring recall, at the end of the tenth word, participants were prompted by the word RECALL followed by a light gray circle on the screen to recall as many words as possible from the previous ten words, in any order. All the signs on the screen during the listening task were the same size and hue to avoid pupil responses caused by visual inputs. Again, the spoken responses were typed down by the experimenter. At the end of each block, participants were asked to rate how effortful the last block was from 1 to 10, with 10 being most effortful. Their subjective ratings were typed down and the program proceeded to the next block. Figure 9 illustrates the procedure. Altogether, each pitch condition was repeated for six blocks, three blocks requiring recall and three blocks not requiring recall (24 blocks in total). Block sequences were randomized for each participant.

Behavioral data analysis

There were three dependent variables (DVs) in the behavioral data: 1) word recognition, 2) word recall, and 3) subjective rating of effort. The first two were binary and the third a score from 0 to 10. Logistic mixed-effect models were fitted to the data in the first two cases (using the Matlab function fitglme with a binomial distribution), and linear mixed-effects models in the third case (using the Matlab function fitlme).

For word recognition, we considered three fixed factors: pitch manipulation (either 4 levels in Exp1, or 2 levels in Exp2), task (i.e. repeat vs recall), and position (01 to 10). We did not expect the position of the word within a list to have much impact on intelligibility, but kept it because of its role on recall (and on the pupil metrics). For word recall (only available in the recall task, not the repeat-only task), there were only two fixed factors: pitch manipulation and position (with linear and quadratic terms to account for primacy/recency effects). For effort rating, the DV existed only per list (not per position) so there were only two fixed factors: pitch manipulation and task.

As for random terms, comparisons between models using a Chi-squared test (based on the change in model deviance) never supported the complexity of random intercepts or random slopes by list. On the other hand, the models always improved with random intercepts by subject. We also considered random slopes by subject for the effect of pitch, task, and position, and this did improve the models but only for certain fixed factors, and not consistently across DVs. Since by-subject random slopes were not helpful for pupil data, we aimed for consistency across the different analyses and restricted random terms to intercepts by subject. This does not yield the most refined model for any given analysis, but it has the advantage of being systematic throughout this article (across the two experiments).

Thus, the final model for word recognition was:

DV ~ task*pitch*position + (1 | subject)

The final model for word recall was:

DV ~ pitch*position^2 + (1 | subject)

The final model for effort rating was:

DV ~ pitch*task + (1 | subject)

All main effects and interactions were determined with a Chi-square test between the model with and without the term in question. As a supplementary analysis, averaged DVs were correlated against the age of participants.

Pupil data analysis

Prior to analysis, pupil traces were cleaned using procedures consistent with previous studies ^37,46,74. Pupil diameter values below 3 standard deviations (SD) of the mean of the whole recording were coded as blinks. Traces within 25 data points before the start and after the end of the blink were cubically interpolated in Matlab, to decrease the impact of the obscured pupil from blinks. Trials that had over 20% of data points counted as blinks were excluded. All valid traces were then low-pass filtered at 10 Hz with a first-order Butterworth filter to preserve only cognitively relevant pupil size modulation ⁷⁵.

For the word recognition phase of each block, (listening and repeating the words), baseline pupil diameters were calculated as the averaged value of the pupil trace over 1 second, just before the start of each word. For the word recall section of the test, baseline pupil diameter was calculated as the mean of the ten previous baseline pupil diameters. The rest of the pupil diameter indices were corrected by the corresponding baselines (i.e., baseline pupil size was subtracted from the trace). This event-related trace was used to analyze pupil size change evoked by word recognition and recall.

Over the presentation of the 10 words of a list, three DVs of pupil response were obtained: pupil baseline, PPD amplitude, and PPD latency. They were extracted within the time window between the onset of word and onset of verbal response, thus excluding any period where the pupil could have fluctuated for articulatory reasons for example. Three linear mixed effects models were fitted, one for each DV, with an identical model of the form:

DV ~ task*pitch*position + (1|subject)

All main effects and interactions were determined with a Chi-squared test between the model with and without the fixed term in question. We also calculated the correlation between each DV and the age of participants.

Finally, to examine the replicability of recently published results using similar experimental paradigms, two additional analyses were conducted, and their methods and results are presented in Supplementary Material 1. Although comparisons of task performance and task-evoked pupillary responses are of central interest in testing our scientific hypothesis, these additional analyses and results can help us to understand better whether previous findings are spurious or can be observed in other independent studies.

Data availability statement

The datasets generated during and/or analysed during the current study are available in the Open Science Framework repository, https://osf.io/74mpy/.

Acknowledgements

We are grateful to all participants for their time and effort. We thank Florian Malaval and Arthur Delage for assistance with running the experiment with native French-speaking participants.

Author contributions

YZ, AL, MD designed the experiment. YZ, AS, AD and MD acquired data and managed participants’ appointments and follow-ups around the study. YZ and MD created scripts and software used in the experiments. YZ, AS, AD and MD analyzed the data. YZ, AS, and MD wrote the manuscript. AL provided critical revisions to the submitted manuscript. All authors contributed to the article and approved the submitted version.

Competing Interests Statement

This research was supported by a grant from the Quebec government (Mitacs Accelerate https://www.mitacs.ca/en/programs/accelerate) in collaboration with an industrial partner Oticon Medical Canada (https://www.oticonmedical.com/) [grant number IT10517]. The funding was issued to Dr. Alexandre Lehmann and Dr. Mickael Deroche, for the postdoctoral work of Dr. Yue Zhang. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The rest of authors declare that no competing interests exist.

World Health Organization. Challenges Facing Ear and Hearing Care. World Rep. Hear. 139–198 (2021).
Livingston, G. et al. Dementia prevention, intervention, and care. Lancet 390, 2673–2734 (2017).
Livingston, G. et al. Dementia prevention, intervention, and care: 2020 report of the Lancet Commission. Lancet (London, England) 396, 413 (2020).
Blamey, P. et al. Factors Affecting Auditory Performance of Postlinguistically Deaf Adults Using Cochlear Implants: An Update with 2251 Patients. Audiol. Neurotol. 18, 36–47 (2012).
Holden, L. K. et al. Factors Affecting Open-Set Word Recognition in Adults with Cochlear Implants. Ear Hear. 34, 342 (2013).
Dorman, M. F. & Gifford, R. H. Speech Understanding in Complex Listening Environments by Listeners Fit With Cochlear Implants. J. Speech, Lang. Hear. Res. 60, 3019–3026 (2017).
Vermeire, K. et al. Quality-of-life benefit from cochlear implantation in the elderly. Otol. Neurotol. 26, 188–195 (2005).
Loizou, P. C. Mimicking the human ear. IEEE Signal Process. Mag. 15, 101–130 (1998).
Shannon, R. V. Multichannel electrical stimulation of the auditory nerve in man. I. Basic psychophysics. Hear. Res. 11, 157–189 (1983).
Zeng, F. G. Temporal pitch in electric hearing. Hear. Res. 174, 101–106 (2002).
Gfeller, K. et al. Musical backgrounds, listening habits, and aesthetic enjoyment of adult cochlear implant recipients. J. Am. Acad. Audiol. 11, 390–406 (2000).
Kong, Y. Y., Cruz, R., Jones, J. A. & Zeng, F. G. Music Perception with Temporal Cues in Acoustic and Electric Hearing. Ear Hear. 25, 173–185 (2004).
Peng, S. C., Lu, N. & Chatterjee, M. Effects of Cooperating and Conflicting Cues on Speech Intonation Recognition by Cochlear Implant Users and Normal Hearing Listeners. Audiol. Neurotol. 14, 327–337 (2009).
Fu, Q.-J., Chinchilla, S., Nogaki, G. & Galvin, J. J. Voice gender identification by cochlear implant users: The role of spectral and temporal resolution. J. Acoust. Soc. Am. 118, 1711–1718 (2005).
Stickney, G. S., Assmann, P. F., Chang, J. & Zeng, F.-G. Effects of cochlear implant processing and fundamental frequency on the intelligibility of competing sentences. J. Acoust. Soc. Am. 122, 1069–1078 (2007).
Moore, B. C. J. The role of temporal fine structure processing in pitch perception, masking, and speech perception for normal-hearing and hearing-impaired people. JARO - J. Assoc. Res. Otolaryngol. 9, 399–406 (2008).
Bernstein, J. G. W. & Oxenham, A. J. The relationship between frequency selectivity and pitch discrimination: Sensorineural hearing loss. J. Acoust. Soc. Am. 120, 3929–3945 (2006).
Jusczyk, P. W. Narrowing the distance to language: One step at a time. J. Commun. Disord. 32, 207–222 (1999).
Soderstrom, M., Seidl, A., Kemler Nelson, D. G. & Jusczyk, P. W. The prosodic bootstrapping of phrases: Evidence from prelinguistic infants. J. Mem. Lang. 49, 249–267 (2003).
Thiessen, E. D., Hill, E. A. & Saffran, J. R. Infant-directed speech facilitates word segmentation. Infancy 7, 53–71 (2005).
Deroche, M. L. D., Lu, H. P., Limb, C. J., Lin, Y. S. & Chatterjee, M. Deficits in the pitch sensitivity of cochlear-implanted children speaking English or Mandarin. Front. Neurosci. 8, 103393 (2014).
Hällgren, M., Larsby, B., Lyxell, B. & Arlinger, S. Speech understanding in quiet and noise, with and without hearing aids. http://dx.doi.org/10.1080/14992020500190011 44, 574–583 (2009).
Nachtegaal, J. et al. Hearing status, need for recovery after work, and psychosocial work characteristics: Results from an internet-based national survey on hearing. https://doi.org/10.1080/14992020902962421 48, 684–691 (2009).
Winn, M. Rapid Release From Listening Effort Resulting From Semantic Context, and Effects of Spectral Degradation and Cochlear Implants. Trends Hear. 20, (2016).
Rönnberg, J. et al. The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances. Front. Syst. Neurosci. 7, 48891 (2013).
Pichora-Fuller, M. K. et al. Hearing impairment and cognitive energy: The framework for understanding effortful listening (FUEL). Ear Hear. 37, 5S-27S (2016).
Rönnberg, J. et al. Hearing impairment, cognition and speech understanding: exploratory factor analyses of a comprehensive test battery for a group of hearing aid users, the n200 study. Int. J. Audiol. 55, (2016).
Sarampalis, A., Kalluri, S., Edwards, B. & Hafter, E. Objective measures of listening effort: Effects of background noise and noise reduction. J. Speech, Lang. Hear. Res. 52, 1230–1240 (2009).
Ng, E. H. N., Rudner, M., Lunner, T., Pedersen, M. S. & Rönnberg, J. Effects of noise and working memory capacity on memory processing of speech for hearing-aid users. https://doi.org/10.3109/14992027.2013.776181 52, 433–441 (2013).
Lunner, T., Rudner, M., Rosenbom, T., Ågren, J. & Ng, E. H. N. Using speech recall in hearing aid fitting and outcome evaluation under ecological test conditions. Ear Hear. 37, 145S-154S (2016).
Frankish, C. Perceptual Organization and Precategorical Acoustic Storage. J. Exp. Psychol. Learn. Mem. Cogn. 15, 469–479 (1989).
Frankish, C. Intonation and auditory grouping in immediate serial recall. Appl. Cogn. Psychol. 9, S5–S22 (1995).
McElhinney, M. & Annett, J. M. Pattern of Efficacy of a Musical Mnemonic on Recall of Familiar Words over Several Presentations. http://dx.doi.org/10.2466/pms.1996.82.2.395 84, 395–400 (1996).
Savino, M., Winter, B., Bosco, A. & Grice, M. Intonation does aid serial recall after all. Psychon. Bull. Rev. 27, 366–372 (2020).
Sares, A. G. et al. Grouping by Time and Pitch Facilitates Free but Not Cued Recall for Word Lists in Normally-Hearing Listeners. Trends Hear. 27, (2023).
Granholm, E., Asarnow, R. F., Sarkin, A. J. & Dykes, K. L. Pupillary responses index cognitive resource limitations. Psychophysiology 33, 457–461 (1996).
Zekveld, A. A. & Kramer, S. E. Cognitive processing load across a wide range of listening conditions: Insights from pupillometry. Psychophysiology 51, 277–284 (2014).
Zekveld, A. A., Kramer, S. E. & Festen, J. M. Cognitive load during speech perception in noise: The influence of age, hearing loss, and cognition on the pupil response. Ear Hear. 32, 498–510 (2011).
Koelewijn, T., Shinn-Cunningham, B. G., Zekveld, A. A. & Kramer, S. E. The pupil response is sensitive to divided attention during speech processing. Hear. Res. 312, 114–120 (2014).
Koelewijn, T., Zekveld, A. A., Festen, J. M. & Kramer, S. E. Pupil dilation uncovers extra listening effort in the presence of a single-talker masker. Ear Hear. 33, 291–300 (2012).
Ohlenforst, B. et al. Impact of SNR, masker type and noise reduction processing on sentence recognition performance and listening effort as indicated by the pupil dilation response. Hear. Res. 365, 90–99 (2018).
Ohlenforst, B. et al. Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation. Hear. Res. 351, 68–79 (2017).
Zekveld, A. A., Kramer, S. E., Rönnberg, J. & Rudner, M. In a Concurrent Memory and Auditory Perception Task, the Pupil Dilation Response Is More Sensitive to Memory Load Than to Auditory Stimulus Characteristics. Ear Hear. 40, 272–286 (2019).
Micula, A. et al. The Effects of Task Difficulty Predictability and Noise Reduction on Recall Performance and Pupil Dilation Responses. Ear Hear. 42, 1668 (2021).
Bönitz, H. et al. How Do We Allocate Our Resources When Listening and Memorizing Speech in Noise? A Pupillometry Study. Ear Hear. 42, 846–859 (2021).
Zhang, Y., Deroche, M. & Lehmann, A. Disentangling listening effort and memory load beyond behavioural evidence: Pupillary response to listening effort during a concurrent memory task. PLoS One 16, e0233251 (2021).
Zekveld, A. A., Koelewijn, T. & Kramer, S. E. The Pupil Dilation Response to Auditory Stimuli: Current State of Knowledge. Trends Hear. 22, (2018).
Singh, L., Nestor, S., Parikh, C. & Yull, A. Influences of infant-directed speech on early word recognition. Infancy 14, 654–666 (2009).
López, S. et al. Vocal caricatures reveal signatures of speaker identity. Sci. Reports 2013 31 3, 1–7 (2013).
Schweinberger, S. R. & von Eiff, C. I. Enhancing socio-emotional communication and quality of life in young cochlear implant recipients: Perspectives from parameter-specific morphing and caricaturing. Front. Neurosci. 16, 956917 (2022).
Racette, A. & Peretz, I. Learning lyrics: To sing or not to sing? Mem. Cogn. 35, 242–253 (2007).
Purnell-Webb, P. & Speelman, C. P. Effects of Music on Memory for Text. http://dx.doi.org/10.2466/pms.106.3.927-957 106, 927–957 (2008).
Savino, M., Bosco, A. & Grice, M. Intonational cues to item position in lists: evidence from a serial recall task. SPEECH PROSODY 708–712 (2017).
Micula, A. et al. A Glimpse of Memory Through the Eyes: Pupillary Responses Measured During Encoding Reflect the Likelihood of Subsequent Memory Recall in an Auditory Free Recall Test. Trends Hear. 26, (2022).
Binns, C. & Culling, J. F. The role of fundamental frequency contours in the perception of speech against interfering speech. J. Acoust. Soc. Am. 122, 1765–1776 (2007).
Meister, H., Landwehr, M., Pyschny, V., Grugel, L. & Walger, M. Use of intonation contours for speech recognition in noise by cochlear implant recipients. J. Acoust. Soc. Am. 129, EL204–EL209 (2011).
Micula, A., Rönnberg, J., Zhang, Y. & Ng, E. H. N. A decrease in physiological arousal accompanied by stable behavioral performance reflects task habituation. Front. Neurosci. 16, 876807 (2022).
Buhusi, C. V. & Meck, W. H. What makes us tick? Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 2005 610 6, 755–765 (2005).
Nittrouer, S., Caldwell-Tarr, A. & Lowenstein, J. H. Working memory in children with cochlear implants: Problems are in storage, not processing. Int. J. Pediatr. Otorhinolaryngol. 77, 1886–1898 (2013).
Baddeley, A. D. Short-term Memory for Word Sequences as a Function of Acoustic, Semantic and Formal Similarity. https://doi.org/10.1080/14640746608400055 18, 362–365 (1966).
CONRAD, R. & HULL, A. J. INFORMATION, ACOUSTIC CONFUSION AND MEMORY SPAN. Br. J. Psychol. 55, 429–432 (1964).
Salamé, P. & Baddeley, A. Phonological factors in STM: Similarity and the unattended speech effect. Bull. Psychon. Soc. 24, 263–265 (1986).
Hopstaken, J. F., van der Linden, D., Bakker, A. B. & Kompier, M. A. J. The window of my eyes: Task disengagement and mental fatigue covary with pupil dynamics. Biol. Psychol. 110, 100–106 (2015).
Ayasse, N. D. & Wingfield, A. Anticipatory Baseline Pupil Diameter Is Sensitive to Differences in Hearing Thresholds. Front. Psychol. 10, 504013 (2020).
Seropian, L. et al. Comparing methods of analysis in pupillometry: application to the assessment of listening effort in hearing-impaired patients. (2017) doi:10.1016/j.heliyon.2022.e09631.
Herrmann, B. & Johnsrude, I. S. A model of listening engagement (MoLE). Hear. Res. 397, 108016 (2020).
Carolan, P. J., Heinrich, A., Munro, K. J. & Millman, R. E. Quantifying the Effects of Motivation on Listening Effort: A Systematic Review and Meta-Analysis. doi:10.1177/23312165211059982.
He, A., Deroche, M. L., Doong, J., Jiradejvong, P. & Limb, C. J. Mandarin tone identification in cochlear implant users using exaggerated pitch contours. Otol. Neurotol. 37, 324–331 (2016).
Meyer, M., Steinhauer, K., Alter, K., Friederici, A. D. & Von Cramon, D. Y. Brain activity varies with modulation of dynamic pitch variance in sentence melody. Brain Lang. 89, 277–289 (2004).
Ilse Lehiste. Some acoustic characteristics of dysarthric speech. ( Basel ; New York : S. Karger, 1965).
Fournier JE. Audiométrie vocale: les épreuves d’intelligibilité et leurs applications au diagnostic, à l’expertise et à la correction prothétique des surdités. (Maloine, 1951).
Kawahara, H. & Morise, M. Technical foundations of TANDEM-STRAIGHT, a speech analysis, modification and synthesis framework. Sadhana - Acad. Proc. Eng. Sci. 36, 713–727 (2011).
Zhang, Y., Malaval, F., Lehmann, A. & Deroche, M. L. D. Luminance effects on pupil dilation in speech-in-noise recognition. PLoS One 17, e0278506 (2022).
Winn, M. B., Wendt, D., Koelewijn, T. & Kuchinsky, S. E. Best Practices and Advice for Using Pupillometry to Measure Listening Effort: An Introduction for Those Who Want to Get Started. Trends Hear. 22, (2018).
Klingner, J., Kumar, R. & Hanrahan, P. Measuring the task-evoked pupillary response with a remote eye tracker. Eye Track. Res. Appl. Symp. 69–72 (2008) doi:10.1145/1344471.1344489.

Competing interest reported. This research was supported by a grant from the Quebec government (Mitacs Accelerate https://www.mitacs.ca/en/programs/accelerate) in collaboration with an industrial partner Oticon Medical Canada (https://www.oticonmedical.com/) [grant number IT10517]. The funding was issued to Dr. Alexandre Lehmann and Dr. Mickael Deroche, for the postdoctoral work of Dr. Yue Zhang. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The rest of authors declare that no competing interests exist.

appendixA.docx
Supplementary material 1: Additional analyses for results replication with previous studies.
appendixB.docx
Supplementary material 2: Age effects on results.

Download PDF

Journal Publication

published 30 Sep, 2024

Read the published version in Scientific Reports →

Editorial decision: Revision requested
07 Dec, 2023
Reviews received at journal
06 Sep, 2023
Reviewers agreed at journal
05 Sep, 2023
Reviews received at journal
29 Aug, 2023
Reviewers agreed at journal
27 Aug, 2023
Reviewers agreed at journal
27 Aug, 2023
Reviewers invited by journal
27 Aug, 2023
Editor assigned by journal
27 Aug, 2023
Editor invited by journal
24 Aug, 2023
Submission checks completed at journal
23 Aug, 2023
First submitted to journal
09 Aug, 2023

You are reading this latest preprint version

Pupillometry reveals effects of pitch manipulations both within and across words on listening effort and short-term memory

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Results

Behavioral data: intelligibility

Behavioral data: recall

Behavioral data: subjective rating

Pupillary data

Pupillary data: baseline

Pupillary data: PPD

Pupillary data: peak latency

Behavioral data: intelligibility

Behavioral data: recall

Behavioral data: subjective rating

Pupillary data

Pupillary data: baseline

Pupillary data: PPD

Pupillary data: peak latency

Discussion

Methods

Declarations

Data availability statement

Acknowledgements

Author contributions

Competing Interests Statement

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1