Eyes Compensate Smile When Wearing Mask

The use of face masks has become ubiquitous. Although mask wearing is a convenient way to reduce the spread of disease, it is important to know how the mask affects our communication via facial expression. For example, when we are wearing the mask and meet a friend, are our facial expressions different compared to when we are not? We investigated the effect of face mask wearing on facial expression, including the area around the eyes. We measured surface electromyography from zygomaticus major, orbicularis oculi, and depressor anguli oris, when people smiled and talked with or without the mask. We found that only orbicularis oculi were facilitated by wearing the mask. We thus concluded that mask wearing increases the use of eye smiling as a form of communication. In other words, we can express joy and happiness even when wearing the mask using eye smiling.


Introduction
How does the use of a face mask in uence our daily communication? Although a part of Japanese culture since the Meiji Era (1868-1912) 1 , face mask usage has only recently become ubiquitous across the world during the outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the virus that causes coronavirus disease 2019 . Unsurprisingly, interest has grown in the physiological and psychological effects of mask wearing 2 , especially as concern regarding interference of communication may contribute to resistance to mask wearing.
The psychological impacts of the face mask on social communication are highly relevant. Facial expression is the one of the most e cient channels to communicate our feelings to others 3 . Among these expressions, smiling is a fundamental facial expression to communicate positive affect and strongly in uences the attention, brain activity, and various behaviors of others 4 . Smiling is achieved by mouth smiling and eye smiling, where the eye smiling was believed to be relevant most when people feel enjoyment 5,6 . This type of smile has been called a Duchenne smile 5,7 . A Duchenne smile has a stronger effect than a non-Duchenne smile (i.e. without eye smiling) on the evaluation of happiness 8 and desirable impression 9,10 . The importance of eye smiling is shown by the nding that botulinum toxin application on the eye muscle diminishes eye smiling and decreases the perceived happiness of the person who received the treatment 11 . Recent studies have argued that facial mask wearing interferes with the recognition of the wearer's emotion and impression 12,13 . To our knowledge, however, there is no investigations into the effect of the mask wearer expressing the emotion. Understanding how the mask wearer expresses their emotion despite the mask is important for as it may provide insights into how mask interference of emotion communication can be overcome.
As the upper face in emotional expression is increased in signi cance when the lower face is hidden by the mask 3,14 , we hypothesized that wearing a mask will enhance eye smiling as the mask wearer attempts to communicate their smile to others. It was widely believed that the Duchenne smile provides a reliable and spontaneous sign of enjoyment because most people are not able to contract orbicularis oculi voluntarily 15 . However, there is evidence that Duchenne smiles can be voluntarily reproduced 15-17 , regardless of whether joy is felt. These ndings suggest that when a person wearing a mask wants to communicate a smile to others, they may try to compensate for the loss of mouth smile information by exaggerating their eye smiling.
To investigate the effect of wearing a mask on facial expression, we used a smiling and speaking task with concurrent facial EMG recording of orbicularis oculi, zygomaticus major activity, and depressor anguli oris. These muscles were chosen as orbicularis oculi is used for eye smilling, zygomaticus major for mouth smiling and depressor anguli oris is used during speech 18 . There are 4 possibilities. First, if our hypothesis is correct and the participants try to overcome the mask intereference using eye smiling, only orbicularis oculi activity will be enhanced when smiling with the mask. Second, if the mask disturbs facial expression visually, but the individual control of the smile related muscles is di cult, both orbicularis oculi and zygomaticus major will be enhanced when smiling with the mask. Third, if the mask disturbs general facial motion physically, the mask wearer may universally exaggerate their facial motions, enhancing both smiling (orbicularis oculi, zygomaticus major) and speaking (depressor anguli oris) to resist the di culty in the motion. Finally, if the mask does not in uence our facial motion, there will be no change in any muscles.

Participants
Twenty females have received remuneration for participation in this experiment (Age: 39.8 ± 11.9 years old), after providing written informed consent. They reported normal or corrected-to-normal vision and no history of psychiatric and neurological disorders. This study was performed in accordance with the principles in the Declaration of Helsinki and approved by the Ethical committee at the Shiseido Global Innovation Center (Approval number: C02121).

Stimuli and procedure
The participants comfortably sat on a chair and were asked to perform 4 behavioural tasks according to the instructions that appeared on the PC screen in front of them (Fig. 1). The rst task was the "photo" task where the participants watched a movie. An experimenter appeared in the movie and pretended to take a picture of the participant, who was asked to smile. The second was the "smiling" task. The participants made their biggest smile that they can immediately after a beep was heard from the PC. Participants were instructed how to make the biggest smile (pull the mouth corner back maximally) and the participants decided whether to open their mouth or not themselves before the experiment. During the experiment, they replicated this type of smile. The third was the "speech" task. The participants spoke the designated 4 sentences (from an ATR 503 phonetically balanced sentences) on the screen. The fourth task was the "free talk" task. The participants talked about 2 themes ("What I'm addicted to these days" and "What I have enjoyed recently") as long as they could up to a maximum of 30 seconds. They were also asked to pretend to talk with a person familiar to them in this task.
Photo and smiling tasks were both conducted twice (once at the beginning and once at the end of the experiment) for each mask condition. All tasks were implemented with or without a surgical mask on their face (Fig. 1). The order of mask condition (with or without mask) was counterbalanced for the repetition (once at the beginning and once at the end of the experiment) and across participants. After the experiment was started, the experimenter left the participants alone in the experimental room as an anticoronavirus infection risk countermeasure and let them initiate the start of the experiment by themselves which they did by pushing the spacebar of the PC. During this time, the participant was recorded on video. Detailed procedures of the behavioural tasks and timing of wearing and taking off the mask were instructed by the in-house developed Python 3.7.3 (Python software foundation, https://www.python.org/) program made with the Psychopy library 19 .

Data acquisition and analysis
Three active bipolar Ag electrodes (AP-C140-020, Miyuki Giken Co., Ltd., Tokyo, Japan) were attached on the face to acquire the electromyography (EMG) from orbicularis oculi, zygomaticus major of the left facial sites, and depressor anguli oris of the right facial sites. The detailed positioning was decided in accordance with published guidelines 20 . Reference and Grand electrodes (AP-C151-015 and MA-C004-015 with general disposal electrode, respectively) were both attached on the forehead and covered with a hairband. To lower the electrode conductance, we applied Ten20 conductive paste (Weaver and Company, Aurora, CO, USA) on the electrode tips and attached a pierced double-sided tape between the body of the electrode and skin. They were also covered with elastic tape. EMG signals were recorded by a 32-ch physiological signal recording device (Polymate V AP5148, Miyuki Giken Co., Ltd., Tokyo, Japan) with a sampling rate of 1000 Hz. 50-Hz noise in the EMG was excluded by a notch lter (48-52 Hz). Then the spiky outlier data over 200 mV was removed. The denoised signal was recti ed and integrated with each 20-ms time window 20 to monitor its time series (Fig. 2-5). The preprocessed EMG signals were divided into epochs of each behavioural task based on the audio signals (beep for the smiling task and voice onset for the photo, speech, and free talk tasks). Due to the selfproceeding experimental procedure, the epochs where the participants failed the procedures were manually checked on the recorded video and excluded from further analyses. This process was done by a rater who took no further role in the data analysis. These failures included failure to comply with the task (e.g. due to failure to hear instructions), misspeaking or misreading. In the photo and smiling tasks, if both trials were rated as successful, the signals from the two trials were averaged. If only one trial was rated as a success and one as a fail, only the successful trial was used in further analysis. The EMG signal was averaged during the tasks and compared between conditions with or without the mask with Wilcoxon signed-rank test with a Bonferroni correction (N = 12 for tasks and measured muscles). Figure 2 shows the group averaged EMG signals recorded on the 3 measured muscles (orbicularis oculi, Zygomaticus major, and Depressor anguli oris) during the photo task. The data in 2 out of 20 participants were excluded from statistical analysis (see Materials & methods). EMG of the orbicularis oculi was signi cantly enhanced by wearing the mask during the photo task (V = 3, n = 18, corrected p < .001, r = .847). The effects of wearing the mask on the other muscles were not statistically signi cant (all corrected p > .250). Figure 3-5 shows the averaged EMGs for the smiling (the data in 2 out of 20 participants were excluded), speech, and free talk tasks (the data in 1 out of 20 participants were excluded) respectively. The effects of wearing the mask on these tasks were not statistically signi cant (all corrected p > .482).

Discussion
During the coronavirus pandemic, wearing a mask has become common and is obligatory in some places. As such, there has been increased interest in the various effects of mask-wearing, including the possible interference effect the mask has on the communication of emotion via facial expression. Concerns regarding possible interference may have exacerbated the resistance to mask-wearing seen in a number of countries. Overcoming the interference during wearing the mask is thus an important goal.
Although mask-wearing blocks the transmission of emotional information from the lower face, we thus hypothesized that wearing a face mask increases eye smiling, compensating for the hidden mouth smiling, allowing communication of smiling.
We tested this hypothesis by measuring face muscle activity using EMG. Speci cally, we looked at the pattern of change in the activity of orbicularis oculi, zygomaticus major, and depressor anguli oris while smiling during mask-wearing. We found that eye smiling (as measured by orbicularis oculi activity) was enhanced when smiling for a photograph during mask-wearing compared to no mask (Fig. 2). In contrast, the muscles related to the mouth smiling (zygomaticus major) and speaking (depressor anguli oris) showed no change when the participants smile (Fig. 2, 3) and speak with wearing the mask (Fig. 4, 5). In other words, when smiling for a photo while wearing a mask, participants increased the use of eye smiling without increasing mouth smiling, consistent with our hypothesis and highlighting the adaptability of human communication.
Recent studies have demonstrated that the emotion of the person wearing a facial mask is less accurately recognized 12,13 . These ndings focus on how is the mask wearer is perceived, while our study focuses on how the mask wearer modi es their behaviour during communication while wearing a mask.
The eye smiling was previously thought to be engaged when a person really feels enjoyment 5,6 and the related muscles (orbicularis oculi) thought to be di cult to voluntarily control 7 . However, recent studies have argued that eye smiling can be intentionally controlled even in the absence of enjoyment. As wearing a mask is unlikely to increase enjoyment, the increase in eye smiling in the current study also support the existence of voluntary control of eye smiling. Note that we could nd a signi cant difference in the EMG activity around the eye during the photo task (Fig. 2) but not during the smiling task (Fig. 3). Krumhuber and colleagues have argued that the occurrence of Duchenne smile (smile with eye smiling) is associated with the spontaneity of the expression 15 . The different results of eye smiling in the photo and smiling tasks may be explained by a difference in spontaneity between the tasks. Smiling as a task in response to a beep stimulus with no person watching is less spontaneous than the photo task.
There are two limitations of this study. The rst is that the participants in this study were all Japanese. As noted before, mask culture in Japan has survived for a very long time 1 . It may be the case that through long exposure to the existing mask culture, our Japanese participants have developed more control over eye smiling that may not be the situation in countries where there is no mask culture. In addition, increased control over eye smiling may also result from the cultural tendency in Japan (and other East Asian countries) to derive emotional information more from the eyes than the mouth, which is the opposite of Western cultures [21][22][23] . This interesting point can be addressed in a future cross-cultural study.
The second limitation is that we did not measure the effect of the enhanced eye smiling during the mask condition of a receiver of the emotional information. It thus remains unclear to what extent the enhancement of eye smiling seen in this study compensates for the hidden mouth smile. Previous studies suggest that masks do impair recognition of emotion 12,13 but this interference effect may be overestimated because these previous studies used photographs where the facial expressions were masked using a digitally added mask. Therefore, there would have been no eye smiling compensation in the mask stimuli. It may be the case that the enhanced eye smiling seen in the current study is enough to compensate, especially if occurring together with the enhancement in gestures and prosody that might be expected in a natural setting.

Conclusion
Smiling motion around the eyes can be enhanced when people communicate their smile while wearing a face mask compared to without a mask. Together with the other modes of communication, such as gestures and prosody 14 that occur in natural settings, we suggest that it possible for humans to maintain communication of emotion even while wearing a mask and that the concern about mask interference of emotion communication may be unnecessary.

Con ict of Interest
The authors report no con icts of interest.

Funding
Details of all funding sources should be provided, including grant numbers if applicable. Please ensure to add all necessary funding information, as after publication this is no longer possible.

Data availability
Due to the con dentiality agreements with the participants, the data in this study are available only at Shiseido co. Ltd.  EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the photo task. The graphs in the left column indicate the time series of EMG signal of these muscles that were integrated with each 20-ms time window (sampling rate was 50 Hz). The solid line is averaged EMG and the coloured area indicates a standard error of the mean (SEM) across participants. Orange and green lines indicate the EMG signal when the participants wore the mask and when they did not, respectively. Right columns indicate comparisons of averaged EMG during a whole task (15 s) when the participants wore the mask (M) and when they did not (NM). Asterisk indicates a signi cant difference in the averaged EMG between NM and M conditions by Wilcoxon signed-rank test with a Bonferroni correction.

Figure 3
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the smiling task (3 s). Plots and colours are the same as in gure 2.

Figure 4
EMG signal measured over the orbicularis oculi, zygomaticus major, and depressor anguli oris, during the speech task (5 s). Plots and colours are the same as in gure 2.