Eyes Compensate Smile When Wearing Mask

doi:10.21203/rs.3.rs-576144/v1

Download PDF

Research Article

Eyes Compensate Smile When Wearing Mask

https://doi.org/10.21203/rs.3.rs-576144/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

The use of face masks has become ubiquitous. Although mask wearing is a convenient way to reduce the spread of disease, it is important to know how the mask affects our communication via facial expression. For example, when we are wearing the mask and meet a friend, are our facial expressions different compared to when we are not? We investigated the effect of face mask wearing on facial expression, including the area around the eyes. We measured surface electromyography from zygomaticus major, orbicularis oculi, and depressor anguli oris, when people smiled and talked with or without the mask. We found that only orbicularis oculi were facilitated by wearing the mask. We thus concluded that mask wearing increases the use of eye smiling as a form of communication. In other words, we can express joy and happiness even when wearing the mask using eye smiling.

Infectious Diseases

Facial muscle

Electromyography

Facial expression

Covid-19

Social communication

How does the use of a face mask influence our daily communication? Although a part of Japanese culture since the Meiji Era (1868–1912) ¹, face mask usage has only recently become ubiquitous across the world during the outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), the virus that causes coronavirus disease 2019 (COVID-19). Unsurprisingly, interest has grown in the physiological and psychological effects of mask wearing ², especially as concern regarding interference of communication may contribute to resistance to mask wearing.

The psychological impacts of the face mask on social communication are highly relevant. Facial expression is the one of the most efficient channels to communicate our feelings to others ³. Among these expressions, smiling is a fundamental facial expression to communicate positive affect and strongly influences the attention, brain activity, and various behaviors of others ⁴. Smiling is achieved by mouth smiling and eye smiling, where the eye smiling was believed to be relevant most when people feel enjoyment ^5,6. This type of smile has been called a Duchenne smile ^5,7. A Duchenne smile has a stronger effect than a non-Duchenne smile (i.e. without eye smiling) on the evaluation of happiness ⁸ and desirable impression ^9,10. The importance of eye smiling is shown by the finding that botulinum toxin application on the eye muscle diminishes eye smiling and decreases the perceived happiness of the person who received the treatment ¹¹. Recent studies have argued that facial mask wearing interferes with the recognition of the wearer’s emotion and impression ^12,13. To our knowledge, however, there is no investigations into the effect of the mask wearer expressing the emotion. Understanding how the mask wearer expresses their emotion despite the mask is important for as it may provide insights into how mask interference of emotion communication can be overcome.

As the upper face in emotional expression is increased in significance when the lower face is hidden by the mask ^3,14, we hypothesized that wearing a mask will enhance eye smiling as the mask wearer attempts to communicate their smile to others. It was widely believed that the Duchenne smile provides a reliable and spontaneous sign of enjoyment because most people are not able to contract orbicularis oculi voluntarily ¹⁵. However, there is evidence that Duchenne smiles can be voluntarily reproduced ^15–17, regardless of whether joy is felt. These findings suggest that when a person wearing a mask wants to communicate a smile to others, they may try to compensate for the loss of mouth smile information by exaggerating their eye smiling.

To investigate the effect of wearing a mask on facial expression, we used a smiling and speaking task with concurrent facial EMG recording of orbicularis oculi, zygomaticus major activity, and depressor anguli oris. These muscles were chosen as orbicularis oculi is used for eye smilling, zygomaticus major for mouth smiling and depressor anguli oris is used during speech ¹⁸. There are 4 possibilities. First, if our hypothesis is correct and the participants try to overcome the mask intereference using eye smiling, only orbicularis oculi activity will be enhanced when smiling with the mask. Second, if the mask disturbs facial expression visually, but the individual control of the smile related muscles is difficult, both orbicularis oculi and zygomaticus major will be enhanced when smiling with the mask. Third, if the mask disturbs general facial motion physically, the mask wearer may universally exaggerate their facial motions, enhancing both smiling (orbicularis oculi, zygomaticus major) and speaking (depressor anguli oris) to resist the difficulty in the motion. Finally, if the mask does not influence our facial motion, there will be no change in any muscles.

2.1 Participants

Twenty females have received remuneration for participation in this experiment (Age: 39.8 ± 11.9 years old), after providing written informed consent. They reported normal or corrected-to-normal vision and no history of psychiatric and neurological disorders. This study was performed in accordance with the principles in the Declaration of Helsinki and approved by the Ethical committee at the Shiseido Global Innovation Center (Approval number: C02121).

2.2 Stimuli and procedure

The participants comfortably sat on a chair and were asked to perform 4 behavioural tasks according to the instructions that appeared on the PC screen in front of them (Fig. 1). The first task was the "photo" task where the participants watched a movie. An experimenter appeared in the movie and pretended to take a picture of the participant, who was asked to smile. The second was the "smiling" task. The participants made their biggest smile that they can immediately after a beep was heard from the PC. Participants were instructed how to make the biggest smile (pull the mouth corner back maximally) and the participants decided whether to open their mouth or not themselves before the experiment. During the experiment, they replicated this type of smile. The third was the "speech” task. The participants spoke the designated 4 sentences (from an ATR 503 phonetically balanced sentences) on the screen. The fourth task was the "free talk" task. The participants talked about 2 themes (“What I'm addicted to these days” and “What I have enjoyed recently”) as long as they could up to a maximum of 30 seconds. They were also asked to pretend to talk with a person familiar to them in this task.

Photo and smiling tasks were both conducted twice (once at the beginning and once at the end of the experiment) for each mask condition. All tasks were implemented with or without a surgical mask on their face (Fig. 1). The order of mask condition (with or without mask) was counterbalanced for the repetition (once at the beginning and once at the end of the experiment) and across participants. After the experiment was started, the experimenter left the participants alone in the experimental room as an anti-coronavirus infection risk countermeasure and let them initiate the start of the experiment by themselves which they did by pushing the spacebar of the PC. During this time, the participant was recorded on video. Detailed procedures of the behavioural tasks and timing of wearing and taking off the mask were instructed by the in-house developed Python 3.7.3 (Python software foundation, https://www.python.org/) program made with the Psychopy library ¹⁹.

2.3 Data acquisition and analysis

Three active bipolar Ag electrodes (AP-C140-020, Miyuki Giken Co., Ltd., Tokyo, Japan) were attached on the face to acquire the electromyography (EMG) from orbicularis oculi, zygomaticus major of the left facial sites, and depressor anguli oris of the right facial sites. The detailed positioning was decided in accordance with published guidelines ²⁰. Reference and Grand electrodes (AP-C151-015 and MA-C004-015 with general disposal electrode, respectively) were both attached on the forehead and covered with a hairband. To lower the electrode conductance, we applied Ten20 conductive paste (Weaver and Company, Aurora, CO, USA) on the electrode tips and attached a pierced double-sided tape between the body of the electrode and skin. They were also covered with elastic tape.

EMG signals were recorded by a 32-ch physiological signal recording device (Polymate V AP5148, Miyuki Giken Co., Ltd., Tokyo, Japan) with a sampling rate of 1000 Hz. 50-Hz noise in the EMG was excluded by a notch filter (48–52 Hz). Then the spiky outlier data over 200 mV was removed. The denoised signal was rectified and integrated with each 20-ms time window ²⁰ to monitor its time series (Fig. 2–5). The preprocessed EMG signals were divided into epochs of each behavioural task based on the audio signals (beep for the smiling task and voice onset for the photo, speech, and free talk tasks). Due to the self-proceeding experimental procedure, the epochs where the participants failed the procedures were manually checked on the recorded video and excluded from further analyses. This process was done by a rater who took no further role in the data analysis. These failures included failure to comply with the task (e.g. due to failure to hear instructions), misspeaking or misreading. In the photo and smiling tasks, if both trials were rated as successful, the signals from the two trials were averaged. If only one trial was rated as a success and one as a fail, only the successful trial was used in further analysis. The EMG signal was averaged during the tasks and compared between conditions with or without the mask with Wilcoxon signed-rank test with a Bonferroni correction (N = 12 for tasks and measured muscles).

Figure 2 shows the group averaged EMG signals recorded on the 3 measured muscles (orbicularis oculi, Zygomaticus major, and Depressor anguli oris) during the photo task. The data in 2 out of 20 participants were excluded from statistical analysis (see Materials & methods). EMG of the orbicularis oculi was significantly enhanced by wearing the mask during the photo task (V = 3, n = 18, corrected p < .001, r = .847). The effects of wearing the mask on the other muscles were not statistically significant (all corrected p > .250). Figure 3-5 shows the averaged EMGs for the smiling (the data in 2 out of 20 participants were excluded), speech, and free talk tasks (the data in 1 out of 20 participants were excluded) respectively. The effects of wearing the mask on these tasks were not statistically significant (all corrected p > .482).

During the coronavirus pandemic, wearing a mask has become common and is obligatory in some places. As such, there has been increased interest in the various effects of mask-wearing, including the possible interference effect the mask has on the communication of emotion via facial expression. Concerns regarding possible interference may have exacerbated the resistance to mask-wearing seen in a number of countries. Overcoming the interference during wearing the mask is thus an important goal. Although mask-wearing blocks the transmission of emotional information from the lower face, we thus hypothesized that wearing a face mask increases eye smiling, compensating for the hidden mouth smiling, allowing communication of smiling.

We tested this hypothesis by measuring face muscle activity using EMG. Specifically, we looked at the pattern of change in the activity of orbicularis oculi, zygomaticus major, and depressor anguli oris while smiling during mask-wearing. We found that eye smiling (as measured by orbicularis oculi activity) was enhanced when smiling for a photograph during mask-wearing compared to no mask (Fig. 2). In contrast, the muscles related to the mouth smiling (zygomaticus major) and speaking (depressor anguli oris) showed no change when the participants smile (Fig. 2, 3) and speak with wearing the mask (Fig. 4, 5). In other words, when smiling for a photo while wearing a mask, participants increased the use of eye smiling without increasing mouth smiling, consistent with our hypothesis and highlighting the adaptability of human communication.

Recent studies have demonstrated that the emotion of the person wearing a facial mask is less accurately recognized ^12,13. These findings focus on how is the mask wearer is perceived, while our study focuses on how the mask wearer modifies their behaviour during communication while wearing a mask. The eye smiling was previously thought to be engaged when a person really feels enjoyment ^5,6 and the related muscles (orbicularis oculi) thought to be difficult to voluntarily control ⁷. However, recent studies have argued that eye smiling can be intentionally controlled even in the absence of enjoyment. As wearing a mask is unlikely to increase enjoyment, the increase in eye smiling in the current study also support the existence of voluntary control of eye smiling. Note that we could find a significant difference in the EMG activity around the eye during the photo task (Fig. 2) but not during the smiling task (Fig. 3). Krumhuber and colleagues have argued that the occurrence of Duchenne smile (smile with eye smiling) is associated with the spontaneity of the expression ¹⁵. The different results of eye smiling in the photo and smiling tasks may be explained by a difference in spontaneity between the tasks. Smiling as a task in response to a beep stimulus with no person watching is less spontaneous than the photo task.

There are two limitations of this study. The first is that the participants in this study were all Japanese. As noted before, mask culture in Japan has survived for a very long time ¹. It may be the case that through long exposure to the existing mask culture, our Japanese participants have developed more control over eye smiling that may not be the situation in countries where there is no mask culture. In addition, increased control over eye smiling may also result from the cultural tendency in Japan (and other East Asian countries) to derive emotional information more from the eyes than the mouth, which is the opposite of Western cultures ^21–23. This interesting point can be addressed in a future cross-cultural study.

The second limitation is that we did not measure the effect of the enhanced eye smiling during the mask condition of a receiver of the emotional information. It thus remains unclear to what extent the enhancement of eye smiling seen in this study compensates for the hidden mouth smile. Previous studies suggest that masks do impair recognition of emotion ^12,13 but this interference effect may be overestimated because these previous studies used photographs where the facial expressions were masked using a digitally added mask. Therefore, there would have been no eye smiling compensation in the mask stimuli. It may be the case that the enhanced eye smiling seen in the current study is enough to compensate, especially if occurring together with the enhancement in gestures and prosody that might be expected in a natural setting.

Smiling motion around the eyes can be enhanced when people communicate their smile while wearing a face mask compared to without a mask. Together with the other modes of communication, such as gestures and prosody ¹⁴ that occur in natural settings, we suggest that it possible for humans to maintain communication of emotion even while wearing a mask and that the concern about mask interference of emotion communication may be unnecessary.

Conflict of Interest

The authors report no conflicts of interest.

Author Contributions

S.O., F.N., N.T., and H.Y. conceived and conducted the experiments. S.O. analysed the results. S.O., H.Y., and K.J.D.K. wrote the paper. All authors reviewed the manuscript.

Funding

Details of all funding sources should be provided, including grant numbers if applicable. Please ensure to add all necessary funding information, as after publication this is no longer possible.

Data availability

Due to the confidentiality agreements with the participants, the data in this study are available only at Shiseido co. Ltd.

Acknowledgements

We thank Keiko Tagai for supporting and coordinating this experiment.

Martin, A. K. T. The history behind Japan’s love of face masks. The Japan times https://www.japantimes.co.jp/news/2020/07/04/national/science-health/japans-history-wearing-masks-coronavirus/ (2020).
Scheid, J. L., Lupien, S. P., Ford, G. S. & West, S. L. Commentary: Physiological and psychological impact of face mask usage during the covid-19 pandemic. Int. J. Environ. Res. Public Health 17, 1–12 (2020).
Mheidly, N., Fares, M. Y., Zalzale, H. & Fares, J. Effect of Face Masks on Interpersonal Communication During the COVID-19 Pandemic. Front. Public Heal. 8, (2020).
Martin, J., Rychlowska, M., Wood, A. & Niedenthal, P. Smiles as Multipurpose Social Signals. Trends in Cognitive Sciences vol. 21 864–877 (2017).
Ekman, P., Davidson, R. J. & Friesen, W. V. The Duchenne Smile: Emotional Expression and Brain Physiology II. J. Pers. Soc. Psychol. 58, 342–353 (1990).
Ekman, P., Friesen, W. V. & O’Sullivan, M. Smiles when lying. J. Pers. Soc. Psychol. 54, 414–420 (1988).
Duchenne, G. B. & de Boulogne, G. B. D. The mechanism of human facial expression. (Cambridge university press, 1990).
Gosselin, P., Perron, M., Legault, M. & Campanella, P. Children’s and adults’ knowledge of the distinction between enjoyment and non enjoyment smiles. J. Nonverbal Behav. 26, 83–108 (2002).
Frank, M. G. & Ekman, P. (1993). Not all smiles are created equal: the differences between enjoyment and non enjoyment smiles. Humor Int. J. Humor Res. 6, 9–26 (1993).
Woodzicka, J. A. Sex differences in self-awareness of smiling during a mock job interview. J. Nonverbal Behav. 32, 109–121 (2008).
Etcoff, N., Stock, S., Krumhuber, E. G. & Reed, L. I. A Novel Test of the Duchenne Marker: Smiles After Botulinum Toxin Treatment for Crow’s Feet Wrinkles. Front. Psychol. 11, 612654 (2021).
Marini, M., Ansani, A., Paglieri, F., Caruana, F. & Viola, M. The impact of facemasks on emotion recognition, trust attribution and re-identification. Sci. Rep. 11, 1–14 (2021).
Grundmann, F., Epstude, K. & Scheibe, S. Face masks reduce emotion-recognition accuracy and perceived closeness. PLoS One 16, e0249792 (2020).
Hombach, S. M. Emotion researcher Ursula Hess explains why a facial expression can be detected when obscured by a face covering. Scientific American https://www.scientificamerican.com/article/from-behind-the-coronavirus-mask-an-unseen-smile-can-still-be-heard/ (2020).
Krumhuber, E. G. & Manstead, A. S. R. Can Duchenne Smiles Be Feigned? New Evidence on Felt and False Smiles. Emotion 9, 807–820 (2009).
Gunnery, S. D., Hall, J. A. & Ruben, M. A. The Deliberate Duchenne Smile: Individual Differences in Expressive Control. J. Nonverbal Behav. 37, 29–41 (2013).
Schmidt, K. L., Ambadar, Z., Cohn, J. F., Reed, L. I. & Schmidt, K. Movement differences between deliberate and spontaneous facial expressions: zygomaticus major action in smiling. J. Nonverbal Behav. 30, 37–52 (2006).
Kurogi, N., Nagai, H. & Nakamura, T. Continuous inaudible recognition of Japanese vowels using features detected at lips shape peaks based on surface electromyography. in 3rd International Conference on Systems and Informatics (ICSAI) 959–964 (2016).
Peirce, J. et al. PsychoPy2: Experiments in behavior made easy. Behav. Res. Methods 51, 195–203 (2019).
Fridlund, A. J. & Cacioppo, J. T. Guidelines for human electromyographic research. Psychophysiology 23, 567–589 (1986).
Yuki, M., Maddux, W. W. & Masuda, T. Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. J. Exp. Soc. Psychol. 43, 303–311 (2007).
Yamamoto, H., Kawahara, M., Kret, M. & Tanaka, A. Cultural Differences in Emoticon Perception: Japanese See the Eyes and Dutch the Mouth of Emoticons. Lett. Evol. Behav. Sci. 11, 40–45 (2020).
Jack, R. E., Garrod, O. G. B., Yu, H., Caldara, R. & Schyns, P. G. Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. U. S. A. 109, 7241–7244 (2012).

No competing interests reported.

Download PDF

Editorial decision: Major revision
05 Jul, 2021
Reviews received at journal
20 Jun, 2021
Reviewers agreed at journal
17 Jun, 2021
Reviewers agreed at journal
15 Jun, 2021
Reviewers invited by journal
14 Jun, 2021
Editor assigned by journal
08 Jun, 2021
Submission checks completed at journal
08 Jun, 2021
First submitted to journal
31 May, 2021

You are reading this latest preprint version

Eyes Compensate Smile When Wearing Mask

Status:

Version 1

Abstract

Figures

Introduction

Materials And Methods

Results

Discussion

Conclusion

Declarations

Conflict of Interest

Author Contributions

Funding

Data availability

Acknowledgements

References

Additional Declarations

Status:

Version 1