Involuntary Markers of Saliency and Surprise Revealed by Oculomotor Inhibition in Response to Auditory Sequences

Our eyes move constantly but are often inhibited momentarily in response to external stimuli. The properties of this Oculomotor-Inhibition (OMI) depend on the stimulus saliency, anticipation, and attention. Previous studies have shown prolonged saccadic inhibition for auditory oddballs; however, they required active counting of the oddballs. Here we investigated whether the OMI response to auditory deviants can provide a quantitative measure of deviance strength (auditory pitch difference) and investigated its dependence on the Inter-Stimulus Interval (ISI), without requesting a voluntary attention to the deviant stimulus. Observers xated on a central xation stimulus and passively listened to repeated short sequences of pure tones that contained a deviant tone either regularly or with 20% probability (the Oddball paradigm). The results showed, as in previous studies, prolonged microsaccade inhibition following the deviant tone. Moreover, the inhibition onset latency was shorter in proportion to the pitch deviance (the saliency effect) and the release was signicantly longer for rare deviants (the surprise effect) as long as the ISI was short (<2.5s). Taken together, these results suggest that OMI provides involuntary markers of saliency and surprise, which can be obtained without the observer’s response.


Introduction
Survival largely depends on one's ability to detect sudden changes in the environment, anticipate upcoming events, and process them optimally. Predictive computations represent one of the fundamental principles of neural processing, and a prediction mismatch may support behavioral complexity and dynamics 1 . Responses to an auditory mismatch can re ect the violations of predictive assessments or adaptation to a repeated stimulus. However, a mismatched response to an omitted predicted signal, for example, is better interpreted as top-down predictive processing rather than simple stimulus adaptation 2,3 . Although many of the studies dealing with auditory deviants used electrophysiology, similar effects can be produced by involuntary eye movement measures 4 .
Involuntary eye movements during xation, including microsaccades, spontaneous eye blinks, and ocular drift, occur continuously; however, the eyes tend to freeze in response to transient stimuli (Oculomotor-Inhibition, OMI) and the latencies of the inhibition onset and release depend on stimulus saliency 5-8 , attention 9,10 , expectations, or surprise 11 . Microsaccades are rapid, small-amplitude saccades that can be executed voluntarily to desired locations, even from memory 12 , but typically they occur involuntarily during xation, with a rate of about one or two per second 13 , and an inhibitory pattern in response to transient stimuli (saccadic inhibition) that has been studied extensively 6,8,10,11,[14][15][16][17][18] . For example, the latency of the rst microsaccade relative to stimulus onset, following inhibition, termed the microsaccade response time (msRT), was found to be sensitive to the contrast and spatial frequency of the stimulus; a shorter msRT occurs with more salient stimuli 6 . Microsaccade inhibition results from a peak of activity at the xation (central) location in the superior colliculus (SC) saccade map 15 , which plays a role in attentive and orienting behaviors and is involved in generating microsaccades 16 . The SC activity re ects a map of the relevant behavioral goals and may correspond to sensory input from different modalities rather than only to a visual stimulus 19 .
In recent studies, microsaccades have been found to be highly informative about cognitive processes. Whereas saliency driven by stimulus properties such as contrast shortens the inhibition, longer inhibition was found for oddballs in a sequence. This was found for a blue patch among frequent red patches 14 , and for a deviant auditory tone in a sequence 4,17,20 . However, these ndings were obtained when observers had to attend to and report the deviant stimulus, possibly re ecting a prolonged inhibition to the attended stimulus rather than a perceptual surprise. In more recent studies from our lab we found preliminary evidence of similar effects obtained in passive viewing without speci cally attending to the oddballs. This was demonstrated for visual oddballs such as a high-contrast patch among low-contrast patches 21 , with similar results obtained with eye blinks 7 and for temporal oddballs (unpredicted intervals) 22 , all showing preliminary evidence of prolonged inhibition for the deviant stimulus. However, prolonged OMI to auditory oddballs without attending speci cally to the deviant stimulus was never demonstrated.
The original purpose of this study was to develop involuntary oculomotor measures of different surprise levels similar to those obtained via ERP 23 to test non-communicating individuals. However, the current study focused on healthy participants. We had two speci c goals: (1) Determine whether the OMI is sensitive to the pitch deviance magnitude (experiment 1), and (2) determine whether it can endure different time scales of rare auditory deviants (experiment 2), all obtained without a request to voluntarily attend to any speci c stimulus.

Participants
Fifty-six observers, 29 females and 27 males, were recruited for Experiment 1 and seventeen, 6 females and 11 males for experiment 2, ages 20-40. One participant was removed from the data analysis of the rst experiment because of excessive blinking and a small number of valid trials with microsaccades. All participants had normal or corrected-to-normal vision and were naïve to the purpose of the study, except for the rst author. The experiments were approved by the Bar-Ilan Internal Review Board (IRB) ethics committee. All participants gave written informed consent and all the experiments were conducted according to the IRB guidelines.

Apparatus
Stimuli were displayed at a distance of 0.6m on a 24-in LCD monitor (Eizo Foris fg2421) running at 1920 x 1080 screen resolution and at a 100Hz refresh rate, using an in-house-developed platform for psychophysical and eye-tracking experiments (PSY) developed by Y.S. Bonneh. All experiments were administered in dim light and the screen background was gray with 50cd/m2 luminance. We used a remote video-based eye tracking system (Eyelink, SR Research), with a sampling rate of 500 Hz, and recorded from a distance of 50-55cm. We chose a 25mm-wide lens in a head-free mode in preparation for the future testing of patients. A task was not used for the same reason and the observers were instructed to xate on the screen center while passively attending to a series of sounds in a "passive attentive" way (see 24 for a similar approach). The stimuli were played through headphones in both experiments. All recordings were done binocularly, with analyses done on data from the left eye. A standard 9-point calibration was performed before each session.

Stimuli and Procedures
Experiment 1: Observers, N=56, passively attended to a series of sounds played through headphones and watched a rapid serial visual presentation (RSVP) of small (~1deg of visual angle), low-contrast, upside down, random Hebrew letters, presented at the center of the screen. One letter was presented for each of the ve sounds in a series. The visual stimuli were not informative regarding the auditory conditions and were used for obtaining a steady xation and for preventing gaze wandering, as well as for obtaining regular OMI, which was modulated by the auditory stimuli. The sound series consisted of rapid sequences of 5 identical tones (70dB SPL, 50ms each, 50ms gap) or they contained a deviant tone at the end of the sequence (a deviant 5th tone). A schematic presentation of the paradigm is shown in Figure 1.
All together there were ve separate conditions, standard trials with ve identical 659Hz pure tones and deviant trials that varied in the frequency of the fth tone with 50Hz steps (709, 759, 809, and 859Hz). The total duration for each presentation of the ve tones and letters was 450ms and the interval between presentations (ISI) was 550ms; therefore, the combined stimulus rate was 1Hz. There was no mixing between trials from different conditions (blocks of 20 trials); however, the blocks were played in random order in three separate short runs. Observers completed a total of 60 trials for each of the ve conditions in three runs (20 trials per condition in a run). Experiment 2: As in experiment 1, observers, N=17, passively viewed and attended to a series of sounds.
To obtain gaze xation and regular OMI, we used a small low-contrast white circle (0.65o in diameter), ashed 50ms before the deviant onset, which remained until the end of the sound series. We applied an auditory oddball paradigm with a rare deviant. There were standard trials with 6 identical 535Hz pure tones and oddball trials with a deviant tone (635Hz) in the third location of the sequence, 200ms after the stimulus onset. (See the schematic presentation of the paradigm in Figure 2.) Stimuli from the Standard and Oddball conditions were played in a mixed order where the number of oddball trials constituted 20% of all trials. The duration of the sound sequence was 550ms and the interval between the sequences (ISI) varied (0.5, 1, 1.5, 2, and 2.5sec) in separate runs. Observers completed a total of 200 trials for each of the ve different ISIs in two short runs per ISI; each included 100 mixed standard and oddball trials.

Microsaccade detection
For the microsaccade detection we used the algorithm introduced by Engbert and Kliegl 25, which is based on the eye movement velocity, and has been implemented in our recent study 26. Raw data were rst smoothed using the LOWESS method with a window of 15ms to optimize microsaccade extraction, especially for noisy recordings 25. Because microsaccades are ballistic movements as saccades show a high correlation between the peak velocity and amplitude, a velocity range of 8o/s-150o/s, and an amplitude range of 0.08-1o were allowed. We also rejected eye movements with a duration smaller than 9ms. Eye blinks were detected as in our previous study 7; we rst de ned periods with zero pupil size, and then extended by estimating the eyes' closed and open times, based on the vertical eye movement that typically precedes the blink 26. Epochs were extracted, triggered by stimulus onset in a range of -0.1sec to 1.1sec relative to this trigger with one epoch per experimental trial. Periods of missing data within an epoch, for example as during an eye blink, were discarded from analysis with an additional margin of 50ms, without discarding the whole epoch. The rejection rate varied across recordings and was typically 5-25%.

Calculation of the microsaccades rate function
The microsaccade rate modulation function was calculated to compute the event-related modulation of eye movements 27 as in 18 and 28; it was calculated for the raw microsaccade onsets (see Figure 3a for an example of a raster plot of microsaccade occurrences) and it is described here brie y. For each epoch the rate was computed by convolving a raw rate estimate of one microsaccade per sample duration with a Gaussian window and with a sigma of 50ms, at the time of microsaccade onset; it was normalized by the number of trials 17. The rates were averaged across the epochs within observers separately for each condition and demeaned by subtracting the observers' mean, then the total average for all conditions and the observers were added. Finally, the mean and standard error were recalculated across observers.

Statistical assessment
To assess the signi cance of the microsaccade rate results, we used a Monte-Carlo cluster-based nonparametric permutation test (as in 17, see also 29) to determine the difference between conditions. We rst looked for a signi cant continuous cluster between the two conditions by performing paired ttests at each time point. Then we randomized the condition labels of the observer's means at each time point and recalculated the group averages to create 1,000 permutations tests, then repeated the rst step. We then computed the p value as the fraction of permutations in which the original test statistic was exceeded by the permuted data.

Microsaccade RT calculation
The Microsaccade Reaction Time (msRT) was calculated for each epoch relative to the stimulus onset in prede ned time windows, as the latency of the last microsaccade in the window of inhibition onset (msRT-last) and the latency of the rst microsaccade in the window of release (msRT-rst), as was done in our previous study 6 and is detailed below. Epochs without microsaccades within the selected windows were excluded from the average, typically around 30-50%. The microsaccade RTs were averaged across the epochs of each condition within observers and demeaned by subtracting the observers' mean, then averaged across observers, and nally, the total average for all conditions and observers was added (Cousineau & Morey's method 30, see also Bonneh et al. 6). This normalization procedure affected only the error bars and did not alter the averages. In computing error bars for the RT values averaged across subjects, we applied the Cousineau method (multiplied by Morey's correction factor (√ (n/ (n−1))), which controls the between-subject variance and allows a better representation of within-subject effects (Cousineau & Morey's method 30, see also Bonneh et al. 6).

Statistical assessment
Statistical analysis of variance (ANOVA) and multiple comparisons post-hoc tests were performed using Matlab 2018b, (One-way in Experiment 1 and Two-way in Experiment 2, the Tukey method). We rst veri ed that the msRT distributions of different conditions come from normal distributions with equal variance. After acquiring the signi cance, we ran two-tailed paired t-tests and applied the Bonferroni correction method for multiple comparisons.

Results
In experiment 1we investigated whether an auditory pitch deviance has an effect on the OMI, which depends on its magnitude. To test this, we extracted epochs triggered by stimulus onset in a range of -0.1sec to 1.1sec relative to this trigger. We rst calculated the microsaccade rate modulation, averaged across observers (N=55) and baseline corrected, and compared the ve conditions: a standard stimulus with ve equal tones (659Hz) and four Deviant stimulus conditions with a deviant tone in the fth location and with different magnitudes (frequency differences) relative to the standard (659+50/100/150/200Hz). Figure 3b shows a typical microsaccade inhibition and release in response to the combined visual and auditory, bimodal stimulus onset (letters and the sound sequence) and a second inhibition in response to the stimulus offset, which describes the different conditions because it starts around the time of the deviant tone. The onset of this second inhibition is measured by the last microsaccade in a window of 100ms and 600ms after stimulus onset and is termed "msRT-last" (see Figure 3c). We found that a larger deviant produced a faster onset of inhibition (F(4,270)=3.07, p<=0.017), which differed from one of our possible initial predictions. The individual scatter plot in Figure 3c (2) shows that most participants had longer microsaccade latencies in the Standard condition compared with 809Hz (+150Hz) Deviant condition, which was found to be signi cantly different in the post-hoc tests (Figure 3d). Figure 3e shows a difference of 809 and 859Hz (+200Hz) Deviant conditions from the Standard, with a histogram comparison and effect size estimation using Cohen's d. Paired t and ranking tests show a signi cant difference (Figures 3e-1, 3e-3) and medium effect sizes were calculated in (Figures 3e-2, 3e-4). Areas under the curve (AUC) of the ROC results are also shown.
In experiment 2 we tested whether the time interval between the stimuli affect the OMI. We estimated a decline in the Oddball effect with longer temporal separation since the sensory memory of preceding item decays and the oddball is de ned by its relationship to the preceding items. The results are shown in Figure 4. Microsaccade rate modulations for an illustration of the Oddball effect, averaged across participants with all ISIs combined (p<= 0.002, Monte-Carlo, permutation test, see Methods) are shown in Figure 4a. As in experiment 1, the rate shows a typical response for stimulus onset and offset but here, the rst inhibition provides more information about the difference between the standard and oddball conditions, determined by the earlier onset of the deviant tone. Here, the visual stimulus was timed 50ms before the oddball to obtain the OMI at the time of the oddball. To quantify the OMI after stimulus onset for the different conditions, we computed "msRT-rst", which is calculated for the latency of the rst microsaccade released from the inhibition in a window of 200-700ms after stimulus onset. Thirteen out of 17 observers had a longer msRT in the Oddball condition (Figure 4b. Trials from all ISIs were combined). The msRT results were analyzed using two-way ANOVA (Figure 4c). Oddball msRTs were found to be signi cantly longer (F(1,160)=4.83,p<=0.029) and msRTs for ISI=0.5sec Standard and Oddball were found to be signi cantly longer (F(4,160)=5.39 p=0.0004, Figure 4d). However, no interaction was found with the ISI factor. We used Bonferroni corrected, 2-tailed paired t-tests to assess the signi cance for multiple comparisons of the Standard and Oddball conditions in each ISI separately. Figure 4e shows the Standard and Oddball conditions in all ve different ISIs (0.5-2.5sec). The results show a signi cant difference between the Standard and Oddball msRTs in ISIs shorter than 2.5sec.
We analyzed the microsaccade rates for different ISIs (Standard and Oddball) combined and found that longer ISIs produced higher microsaccade rates (shown in Figure 5a). We then calculated the "microsaccade hit rate" (denoted here as "mshit") corresponding to the percentage of trials with at least one microsaccade occurring within a time window of 200-700ms after stimulus onset. We found that (1) Oddball trials had signi cantly higher microsaccade hit rates at that window for all ISIs (Figure 5b

Discussion
In the current study we conducted two experiments with auditory pitch deviance and measured the OMI response. We found that when the deviant was frequent and therefore predictable, microsaccade inhibition onset was faster as a function of the deviant size (higher pitch) and when it was rare, the inhibition was prolonged. This was achieved without the observer's response. Next, we will discuss the different aspects of these ndings and their interpretation.
Auditory deviance and OMI: salience vs. surprise Given that an auditory oddball was found to increase the period of saccadic inhibition, i.e., postpone its release, at least when a task was involved 4,17 , one would expect that this inhibition will be prolonged with a larger deviance. On the other hand, since the OMI is known to be faster and shorter with the saliency of the stimuli, e.g., for higher visual contrast 6 , one would expect faster and shorter inhibition with a larger deviance that appears perceptually more salient.
In experiment 1, the stimulus was a repeated short sequence of 5 identical tones, or it contained a fth deviant tone. The deviant was xed within a condition and varied between conditions; in this way, the deviant was frequent and totally predictable. We found that the OMI was sensitive to the auditory pitch deviance and was affected by its magnitude. In response to a larger deviance, the OMI was faster (started earlier) as evident by the difference in the rate modulation function (Figure 3b) and by the earlier inhibition onset around the time of the deviant tone presentation (Figure 3c). Because the deviant was predictable, the results resemble those of the OMI in response to visual contrast stimuli 6 that show a faster inhibition onset for higher contrast. Here, the effect of the deviant tone did not stem from an unpredictable change or surprise, but instead from its contrast with the 4 preceding standard tones, hence, the similarity to the effect of visual contrast. We therefore can conclude that the OMI measured in this experiment re ects the perceived stimulus saliency.
The results of this experiment could be explained by referring to two early processes of change detection in the auditory modality, originating from the auditory cortex and biasing attention away from the common stimulus. Stimulus Speci c Adaptation (SSA) and Mismatch Negativity (MMN) are not entirely dissociated, and some studies suggested that SSA provides a neuronal correlate of MMN 31,32 . Stimulus Speci c Adaptation (SSA) re ects the habituation to a recurring stimulus, spanning several time scales ranging from milliseconds to tens of seconds 33 . The evoked potential (ERP) Mismatch Negativity (MMN) re ects the brain's response to a sudden change in stimulus, peaking at about 150ms [34][35][36] . Based on these known processes, we considered the following explanation of our results: Early inhibition onset indicates preparation and is associated with temporal anticipation due to the paradigm's design. A higher sensory saliency of the deviant sequence is caused by habituation of the repetitive reference tone (SSA) and a fresh response to the deviant tone (depending on its deviance 37 ), resulting in faster inhibition onsets. In addition, the mismatched deviant tone signals a prediction error relative to the size of the difference, leading to a better adjustment of a temporal model 38 .
In experiment 2 we used an oddball paradigm, where an infrequent sequence of tones was presented randomly among 6 repeated identical tone sequences at a ratio of 20/80. The results showed that a rare auditory pattern induced a signi cant surprise response, with a prolonged OMI as in previous oddball studies 4,17,20 . With a long inter-stimulus interval (ISI) of two and a half seconds, this effect became nonsigni cant (p=0.05, 2-tailed paired t-test) (Figure 4e). This might be associated with a reduction in alertness due to the slow pace of the experiment, together with the lack of any attentive task. The results indicate a surprise response re ected by prolonged inhibition; however, it could also be associated with early change detection processes as well as an additional top-down re-direction of attention. A late ERP component is associated with attention shifts; P3a is a subcomponent of the P300 complex, which re ects the top-down response to violations of expectations and decision making 39 . It occurs 150ms later than MMN peaking, around ~300ms and requires a detection task. Since microsaccade inhibition covaries with spatial attention 25,40,41 and it was reported to be induced by the allocation of spatial attention to the xation location in the visual eld 15,42 , we believe that top-down and bottom-up attention re-orientation mechanisms account for the prolonged inhibition.
Note that due to the limited constraints derived from our results, this is not the only reasonable explanation and alternative explanations could also be considered; however, they are left for future work.

The effect of Inter Stimulus Intervals on OMI
We also observed a signi cant increase in the microsaccade rate as a function of ISI (Figure 5a). This could be explained by reduced alertness and it could result from reduced inhibition in the longer ISIs due to a lower stimulus rate. The link between alertness and microsaccade inhibition was demonstrated, for example, in the nding of reduced inhibition in ADHD in a continuous performance task, which was recovered by administering a stimulating medication 10 . It was also reported that higher attentional loads, as in the shorter ISIs with increased alertness, are associated with a lower microsaccade rate 43 . An alternative explanation may be related to the microsaccade preparation time; there is less time to prepare in the shorter intervals, resulting in fewer microsaccades. Both explanations could be supported by the highly signi cant longer msRTs in the 0.5sec ISI condition (F(4,160)=5.39 p=0.0004).

Comparison with previous Oddball studies
Previous OMI studies of auditory oddballs 4 involved a task that could have in uenced the results by generating an attention effect. For example, our preliminary results from a serial dependency study, in which participants were asked to count a colored patch from a group of red and green patches, showed a signi cantly longer OMI for the attended stimuli 21 . A task such as counting the oddballs 4 involves additional processing time to hold the current number of oddballs within working memory (WM) 44 and to make a decision involving target discrimination. Thus, it could have prolonged the saccadic inhibition for targets regardless of whether or not these targets were oddballs. In our study, the participants were asked to attend to all sounds without any request to pay speci c attention to the oddballs in a passive attentive way. We show here, for the rst time, the OMI effects for auditory oddballs in a passive attentive paradigm (see 23,24 for a similar passive-attentive paradigm). However, these OMI effects appear smaller than those obtained with attended stimuli 4 or in response to visual stimuli as a function of contrast 6 .
Our study implements an auditory oddball paradigm similar to that of Bekinschtein et al. 23 , which measured the ERP markers of violations of auditory regularities, either "local" in time, within a single trial (similar to our exp1), or "global" across trials of several seconds (similar to our exp2). Their Local-Global paradigm suggests the existence of a hierarchical organization consisting of at least two levels of perceptual prediction mechanisms: (1) an early mechanism, re ected in the MMN signal, which is effective only in a limited time window for changes that are "local" in time 45 , and (2) a later, more distributed predictive mechanism, re ected by P3b (a second subcomponent of P300) response to more "global" violations of expectations 2 . They report a global effect as a marker of awareness for a rare auditory pattern with an ISI of ~1.5 seconds, measured by P3b when participants were asked to count the oddballs. In contrast, when participants were engaged in mind-wandering or in an active visual target detection of letters, the P3b magnitude for the surprise sounds decreased dramatically 23 . Thus, it follows that in this study the P3b signal could have resulted from counting rather than as a marker of predictive violation because P3b is also related to context updating and is associated with memory operations 46 as holding the number of oddball occurrences in working memory. When we compared our results to this ERP study 23 , we found both similarities and differences. Unlike Bekinschtein et al., we did not nd an OMI effect for the reversed combination of a global standard (AAAAB) and a global deviant (AAAAA), which implies a strong contribution of early mechanisms to the oddball OMI effect (data not shown). Our participants reported being aware of the oddballs when asked after the experiment; however, their level of engagement and its contribution to the OMI are unknown. It is therefore impossible to distinguish between the contribution of an automatic change detection process and a higher-level predictive mechanism.

Summary And Conclusions
The Oculomotor Inhibition (OMI) that we measured for microsaccades was found to be sensitive to auditory deviance and its magnitude (a marker of saliency); it exhibited prolonged inhibition for rare deviants (a marker of surprise). Moreover, more microsaccades occurred with longer intervals between stimulus presentations (ISI) and with shorter latencies. These results were obtained with a passive attentive paradigm and without an active task that would have directed attention to the oddballs.  Figure 1 Schematic illustration of the trial sequence of experiment 1. There were auditory and visual stimuli of ve conditions with a fth tone frequency difference. Trials from the different conditions were played in separate blocks and there was no mixing between conditions.

Figure 2
Schematic illustration of the trial sequence of experiment 2. Oddball trials with a deviant tone were interleaved within the standard trials with a ratio of 20/80. The group average msRT for the Standard and Oddball conditions for different ISIs (0.5-2.5sec). Error bars denote 1SE across observers. Multiple comparisons were run using 2-tailed paired t-tests (Bonferroni correction was applied). A signi cant difference in msRT was found for the rare oddball effect in the short ISIs (ISI<2.5sec). In all conditions, the msRT was calculated in a time window of 200-700ms poststimulus onset.