A New Method to Better Measure and Interpret NonVerbal Communication in Patient-Therapist Interactions

Objective(s): To develop a novel method of examining facial affects in psychotherapeutic settings and interpreting the subliminal microexpressions and emotions they represent. Results: Therapy sessions are videotaped, and verbal content is evaluated with computer assistance from ATLAS.ti. Visual content is screened for facial action units/micro-expressions using the Facial Action Coding System (FACS) and reevaluated using the Emotional Facial Action Coding System (EmFACS) in order to discern the underlying emotions. Working Alliance Inventory scores are compiled to assess their impact on the quality of the therapeutic relationship. Reliability of facial action unit coding is ensured by a training course and an independently evaluated standardized test at Innsbruck University. Interrater reliability was excellent (Cohen’s κ > 0.80). Application of this method on a small study population to measure non-verbal communication and affects demonstrated its feasibility and usability. We believe this methodology to be widely implementable. Its application in psychotherapy may provide greater insights in investigations into important aspects of therapeutic interaction (e.g. verbal/non-verbal communication between clinicians and patients in psychiatric contexts, quality of psychotherapeutic relationships, etc.), and may be exploitable in clinical psychotherapeutic settings to better understand mechanisms of change in therapy.


Introduction
Process-oriented research is an important part of modern psychotherapeutic research. Nevertheless, there remains a long way to go in order to fully understand psychotherapeutic processes in the treatment of mental illness. In particular, the investigation of so-called therapeutic micro-processes, small but essential aspects of the patient -therapist interaction (especially facial micro-expressions), could bring promising insight into the mechanism of psychotherapy, as well as allow identi cation of speci c factors essential to successful psychotherapy, regardless of the types of psychotherapeutic methods used.
Studies show that 60-80% of all communication (in treatment) is nonverbal (1,2). Many nonverbal events are unconscious and often convey a patient's emotional and psychological state in a way that verbal communication cannot (3). In any given interpersonal interaction, a great number and variety of facial expressions are normally present (4); unfortunately, many of the visible expressions in conversations of two or more people are often inauthentic and very consciously controlled. Microexpressions, on the other hand, can neither be controlled nor be voluntarily displayed, and thus offer a truer view into the genuine feelings and emotions of a person. Discovered in the late 1960s by Paul Ekman, micro-expressions are de ned as facial expressions of brief duration, lasting only for one-quarter to one-half of a second (5), and can be understood as either repressed or unconscious expression of emotion (2, 6). Ekman's research led him differentiate and name seven universal or basic emotions, expressible in the form of micro-expressions: Happiness, Surprise, Anger, Sadness, Fear, Disgust, and Contempt. In the context of psychotherapeutic interactions, the fact that micro-expressions are signals perceived unconsciously and can in uence the patient-therapist relationship dynamic makes their investigation worthwhile.
What's more, therapy sessions with higher levels of nonverbal micro-affectivity have been rated as having more impact on patient outcome (7). If that is the case, it shows that common methods of analysis between cognition and affect might be incomplete, as they do not capture the speechless forms of affective expression. In addition, subconscious interaction patterns are implemented trough nonverbal signals. If the therapist is able to resist this relationship offer verbally as well as nonverbally, then pathological relationship patterns may be prevented, which in turn may increase the probability of successful treatment (8).
In this light, it seems promising to investigate the association between verbal psychotherapeutic interventions and the involuntary facial micro-expressions provoked by them (9), especially as it pertains to the working alliance, as this may allow us to better evaluate the effectiveness of certain interventions in therapeutic settings. Here below, we describe our novel method of how to measure and evaluate these micro-processes, with some preliminary ndings obtained by applying them on a small sample study population.

Study Population
Our methods were applied on 20 willingly participating adult patients (10 males, 10 females; mean age =

Video Recording
Videotaped therapy sessions were held between the patient and the therapist alone. Both therapist and patient were recorded with separate video cameras, providing front views of each, using two high de nition 4 k cameras, each recording sixty frames per second. This is necessary in order to be able to detect the very subtle facial movements producing micro-expressions. In order to randomize the material, the second ve minutes of each recorded session are analyzed with a split-screen technique.

Verbal Content Analysis
Following videotaping, sessions are transcribed, and their verbal content evaluated ("content analysis") according to the Psychodynamic Intervention List (PIL) (10), an instrument used to identify and rate psychodynamic verbal techniques, with computer assistance from ATLAS.ti. To increase con dence in the ndings, self-and observer-rated measures are combined before and after the sessions.
ATLAS.ti is software developed for systematic analysis and structuring of unstructured data, not only in text-form, but also multimedia les, such as videos. This was very conducive to this project as all four raters were able to feed videos of recorded sessions into the software and then analyze the transcripts while watching the playback, thus allowing the inclusion of atmospheric and contextual facets of the interaction.
This analysis was performed in order to capture verbal statements within interactions and categorize them, with a focus on identifying potential patterns in patient or therapist micro-expressions in association with certain subjects or types of verbal interactions. Through this process, qualitative data can be transformed into quantitative units, allowing for a reduction of complexity and thus facilitating comparisons.

FACS + EmFACS Analysis / Coding
The second phase of analysis consists of analyzing the visual content for action units using the Facial Action Coding System (FACS) (11). FACS is a coding system, associating facial expressions with socalled Action Units (AU) and Action Descriptors (AD). AUs are de ned as the basic actions of individual muscles or groups of muscles, whereas ADs are described as unitary movements and actions of several muscle groups working together.
The coding is performed following a complex set of rules, outlined in (12). In essence, the authors divide the face into different parts and different rating units. In total, there exist twenty-ve different action units that can emerge individually or in combinations, with differing positions of the head and eyes, and varying degrees of deceptive activity. Additionally, the intensity of the displayed action units is scored.
The intensity scale ranges from A-E, with A indicating minimal trace activity of the action unit, up to E, which indicates maximal intensity.
To perform the FACS coding, the videos of therapeutic sessions were assessed with split screen after having identi ed the different types of interventions and rating them using PIL. First, the videos were displayed together, and then separately at original speed. Following the guidelines of Ekman et al. Following that, the coded action units are then reevaluated using the Emotional Facial Action Coding System (EmFACS) (13) and Cohn-Kanade (CK+) datasets in concert (14). EmFACS is simply a system for classifying facial expressions that have been previously coded into so-called action units by the Facial Action Coding System. Using the FACS codes, the EmFACS instrument partitions the scored facial movements into groups that representation affective display. The coding must only be performed by FACS-certi ed raters, as according to Ekman and his group, only trained raters are quali ed to perform EmFACS rating. We complemented our evaluation with the very popular Extended CK + Dataset (14), since this dataset is generally considered as being more comprehensive than the basic one that EmFACS traditionally uses, with many more AU combinations and associated emotions available.

Validation
The reliability of all coding was ensured by a training course at the Study Group of Prof. Eva Bänninger-Huber from Innsbruck University, followed by a standardized nal test, which was independently evaluated by the team of Paul Ekman. Interrater reliability of the nal test was very good (Cohen's κ > 0.80).
Raters conducting the content analysis and the analysis of the Facial Action Coding System and EmFACS were all blinded to the patients' diagnoses, as well as the therapists' theoretical background and experience.

Working Alliance
Finally, the results of the Working Alliance Inventory were evaluated via the Working Alliance Inventory-Short Revised (WAI-SR), which measures the quality of the patient-therapist working relationship by assessing three key aspects: (a) agreement on the tasks of therapy, (b) agreement on the goals of therapy and (c) development of an affective bond. (15). Studies revealed that working alliance quality has a signi cant impact on treatment outcome and can even be used as a predictor of success (16)(17)(18).

Statistical Analysis
Statistical analyses were performed using the IBM SPSS Statistics 19 software (IBM Corp., Armonk, NY, USA). For all analyses, the signi cance threshold was de ned as p ≤ 0.05.
Differences between therapists and patients with respect to facial affects analysed via FACS were assessed with Mann-Whitney´s U test. The standardized mean group difference, Cohen´s d, and the conventional benchmarks for interpretation (small effect: r ≤ .10, medium effect: r ≤ .30, large effect: r ≤ .50; [33]) were applied.

Results
Overall, the group of patients displayed signi cantly fewer facial affects than the group of therapists (Mann-Whitney-U = 146.0, p = 0.023, d Cohen =0.72). However, there was no signi cant difference regarding the variety of different facial affects between therapists and patients (Mann-Whitney-U = 199.5, p = 0.304, d Cohen =0.308). Table 1 shows the frequency of each basic affect in patients and therapists. The most frequently occurring affects in patients were disgust and surprise; in therapists, contempt and surprise. Since in both groups, fear, anger, and sadness affects were only marginally present, for simplicity these three were grouped together under term "else".

Discussion
The methodology outlined in this paper was developed and re ned at Vienna General Hospital, in what could be considered a typical social psychiatric ward. This department provides everyday psychiatric hospitalization services to those who need them, offering coverage to the roughly 200,000 Austrian citizens living within its service radius. The patients encountered there on a daily basis are extremely diverse in many aspects, as much from a demographic point of view as from a medical one. Many times, this diversity poses great challenges in communicating with, handling, and/or treating these patients.
Often times, for example, a patient may not speak the same language as the treating personnel or staff, or perhaps the patient's medical condition (psychiatric or otherwise), level of cognitive function or even literacy hinders verbal communication. In other scenarios as well, for whatever reason, there may be concealment or untruthfulness (either wilfully or inadvertently) in verbal interactions or in responding to typical clinical self-report questionnaires items. In such cases, the patient-clinician relationship will likely be less than ideal, rendering treatment di cult, and at times, perhaps even dangerous. Tools that would allow monitoring patient affects more closely and accurately at all phases of a patient's therapeutic experience (from admission and during treatment, to discharge) would be a signi cant advance, potentially improving patient outcomes and security of all parties involved in treatment.
Overall, the methods developed and presented in this work are extremely feasible, adaptable, and powerful tools that may prove very useful across a wide variety of clinical contexts: treatment progression monitoring, to be sure, but also for enhancing clinical training of practitioners and the patient-clinician working relationship. In so doing, this may serve to improve psychotherapeutic treatment outcomes in the long-term.

Limitations
Some limitations should be noted. Firstly, EmFACS can only detect very primary or basic emotions, so at times the instrument fails to distinguish nuances of displayed primary affects, even when complemented with the CK + dataset. This task is thus left to the interpreters of the EmFACS results, with a high degree of subjectivity involved in the evaluation. In addition, if one is limited to EmFACS, it is di cult to discern what complex and multifaceted dimensions or meanings any discernible emotion might have for the interacting partners, especially when it comes to unconscious understanding. Finally, the complexity and meticulousness of the rating methods makes it quite demanding (time-consuming), as evaluating even one minute of a video interview corresponds to roughly ve hours of work, in order to process and rate the material contained therein.
Ideally, the validity of the coding of emotions obtained through EmFACS should be veri ed by a comparison of its ndings against those obtained through other means (preferably one considered as being a "gold-standard"). Further re nement of these techniques is urgently needed to gather real-world data and address the major limitations we have identi ed at present, in order to increase the scope of their applicability as well as reduce the time and resource demands. Availability of data and material: The datasets generated and analyzed during the current study are not publicly available due to reasons of con dentiality according to the ethics committee of the Medical University of Vienna but are available from the corresponding author upon reasonable request.

Abbreviations
Competing interests: All authors declare that they have no con ict of interest.