Given the mounting worldwide mental health impact reported recently in Oct 2020 by Lancet’s editorial [31], the focus is on designing an AI approach that can address all the 3 dimensions of the challenge. A scalable approach to impact millions in a timely intervention in a privacy-safe way is the focus. BERT activated GPT2 for the generation of short narratives for seeding healthy thoughts is demonstrated. The sample narratives generated by AI are shown to look similar or as good as narratives generated by human counselors. To offer CBT like self-help to correct one’s outlook towards a situation, a conditional language model is created using transfer learning. A human-like text generation and the synthesis of conditioned language is demonstrated. The sample outputs shown in Table 4 show the sentances generated by AI is as good as one spoken by humans, thus showing the feasibility of employing AI to scale mental health counselors’ impact. Mental health experts can scale their productivity by training an AI, which can assist their patients in their absence. This is important given the need to support millions of families, given the reported shortage of capacity in mental health services [31, 32]. The use of such AI also ensures both early and timely activation of therapy. Further, a NLU activated NLG based therapy, ensures activation at the right time. A Human-Computer Interaction approach of a proposed mental health keyboard with an on-device AI inference ensures 24x7 mental health assistance for millions of individuals. The on-device AI inference supports the privacy and protection of one’s personal information. Comparing the performance of different GPT2 model shows a small-sized GPT2 with 224 Million parameters is a choice for widespread deployment on-device on smartphones. Future research can extend the performance on smartphones using DistillGPT.
Among the 3 results, Result #1 represents the primary result of this paper. While novel ideas are presented in Result #1 and Result #3, the paper doesn’t make any novelty claims for Result #2. The two contributions and the three results are specified in Table 2.
2.1 Result #1: Novel idea of NLG in therapy to avert the crisis: Design of a AI for early intervention in mental healthcare for millions while ensuring safety of sensitive info
Challenges addressed by this result:
- WHO (World Health Organization) reported the need for action on mental health [3]. The challenge is to design mental health therapy to help the individual look at a given situation from a different perspective, in order to lead to mental wellness [8]. As an illustrative example, in a situation of a person losing a job during the COVID-19 pandemic, the AI should help the individual develop a positive inner belief [6]. The challenge is to design an AI based therapy by applying the state of art advancements in Deep Learning in NLG.
- The challenge is designing an approach that can meet the demands of a pandemic scale mental health solution [2]. The need is a scalable solution to serve millions of families with 24 hours of continuous support for each of the families.
- Early intervention is an implementation challenge for 21st century mental health care as per JAMA [44]. Early intervention can be defined as diagnosis and treatment at the earliest possible point, even presymptomatically [44]. Today’s challenge in mental healthcare is that treatments are typically deployed late and without the strategic goal of reducing the progression of the illness, as per JAMA [44].
- Privacy concerns as mental health counseling often involve sharing of sensitive personal information with the therapist [24].
- There is a reported shortage in the capacity of mental health services as per WHO’s assessment in year 2020 [31]. Given the pandemic, the need for tools to significantly multiply mental health professionals’ productivity is essential [24]. Going forward, a mental health therapist should be able to care for a significantly larger number of patients.
Methods & Discussions:
a) A response to the call to action to avert the forthcoming crisis:
The proposed AI is designed as a response to WHO’s call to action on global mental health [3, 31]. As per WHO, almost 1 billion people suffer from a mental disorder [38]. Around 1 in 5 of the world’s adolescents suffer from a mental disorder, as per WHO facts [38]. The economy loses US$ 1 trillion every year in productivity because of depression or anxiety [38]. The Lancet editorial in Nov 2020 reports the monumental effects of COVID-19 pandemic on mental health [40]. The editorial [32] raises the following concern:- It is unclear how the world will deal with this forthcoming crisis, as the capacity of mental health services to respond in such a large scale doesn’t exist today. Hence the need for a scalable approach like the one proposed in this paper becomes significant.
b) The vast untapped potential of Transformers based AI architecture for mental health therapy.:
Though Deep Learning research is progressing at an amazing pace, the opportunity to apply Transformers architecture [28] powered text generator [25] to improve mental health is not yet explored in the existing literature. Given the forthcoming crisis, it is imperative to explore the feasibility of employing the latest advances in AI for mental health. Inorder to avert the titanic crisis, this untapped potential is explored in this paper. This paper presents a novel idea at the intersection of CBT [17] & AI.
As per the May 2020 expert review [4, 5], the opportunity to apply a powerful deep neural network of the order of 100 Million neural network parameters for mental health therapy solutions to avert the looming mental health crisis is less explored in the literature. The benefit of such a neural network is human-like performance in language modelling [21] . The ability to fine-tune the models using transfer learning technique [13] allows the computer-based generation of language that is as close as possible to human counselors [6, 24]. One reason is progress in large NLG models such as GPT2 (Generative Pretrained Transformer 2) [25] happened very recently in year 2019. A search of “GPT2 and Cognitive Behavior Therapy” in Google Scholar yields no result.
GPT-3 [43] introduced in year 2020 is too large for on-device AI inference on smartphones. Unlike GPT-2, GPT-3 is only accessible as a cloud API. So GPT-3 can’t meet the privacy requirement of on-device AI inference. Hence GPT-3 is not suitable for consideration for this mental health challenge.
Using GPT2 to help patients practice Cognitive Behavior therapy(CBT) to change their negative view – all in real-time with privacy safety is a new idea. Further, given the scale of the problem, the therapy has to proactive rather than user-initiated. The idea of a timely health intervention in a privacy safe approach for millions of families by novel application of GPT2 is made possible now due to the advancements in AI, and this paper proposes this idea & demonstrates with a working prototype.
c) Digital interventions in mental health approaches such as Cognitive Behavior Therapy (CBT):.
Experts have called for the involvement of multi-disciplinary research [2] to avert the forthcoming crisis. While this paper’s primary highlight is on Deep Learning applications, this paragraph introduces mental health therapy concepts such as Cognitive Behaviour Therapy (CBT).
CBT is a popular form of mental health therapy. Cognitive Behavior Therapy (CBT) [17] is psycho-social intervention that aims to improve mental health. Cognitive model of depressed individuals self-construct such a negative view of himself [6]. Beck’s “Cognitive triad” [7] enables the patient to correct their view of the world [8]. By changing unhelpful or inaccurate thinking, Cognitive therapy equips individuals to practice more flexible ways to think to overcome the cognitive distortion [6]. The JAMA article [9] concluded the effect of early intervention using Cognitive therapy. But the number of people who need help is multiple order of magnitude higher due to the pandemic, hence JAMA article [1] calls for creative thinking in treatment. The Lancet Psychiatry position paper calls for Digital interventions [2]. This delineates the potential of applying Deep Neural network based “Digital Cognitive therapy”. From the perspective of an interdisciplinary researcher, the opportunity for AI-based CBT is proposed in this paper. The concept idea of AI based CBT is offered in Figure 2. As shown in the figure, the circle of thoughts and inner beliefs of an individual can be ‘influenced’ by helping the person to change the way he ‘perceives’ the situation. The figure also illustrates the example where AI helps the individual to perceive his loss of a job from the viewpoint of his strengths. The video in the URL further clearly articulates how AI is able to sow healthy thoughts.
d) AI for transforming Early Intervention in mental health care:.
Early intervention is an implementation challenge for 21st century Mental Healthcare as per JAMA Psychiatry [44]. Early intervention is significant in many healthcare settings. The same applies for mental healthcare too. Early intervention is giving care at the earliest possible point or pre-symptomatically [44]. In contrast to visiting counseling centers physically, AI allows for CBT inspired self-help almost instantly. Though an appointment with a human counceller may take weeks, especially with the reported shortage of capacity of mental health services, an AI based intervention can happen almost instantly. AI therapy enabled smartphone can bring care to the individual in the early stage of depression. A NLU activated therapy can turn on the therapy at an appropriate time as it detects if the person needs help. This ability for a proactive AI based CBT presents a breakthrough. AI can thus provide a solution to the 21st century challenge of early intervention in mental healthcare. The idea of early intervention in mental health by creatively applying AI can yield breakthrough advancements in the mental health of a person.
Result & Methods: Novel application of GPT2 for CBT to address the 3 challenges
- Highlights of Result #1 :
- In response to WHO and experts’ call for action to avert the looming mental health crisis, this result contributes an AI-based solution.
- This result contributes a solution that addressed the 3 dimensions of the challenge: scale for millions, timely intervention & privacy safety.
- As of 8th Jan 2021, there is a lack of research publications on applying state of the art AI models such as GPT2 for mental health therapy. To the best of our knowledge, this paper is the first to apply GPT2 for CBT.
- Novel application of GPT2 for mental health therapy solution is proposed.
- The proposed concept of AI aided therapy is articulated in Figure 2.
- The power of Transformers based NLG architecture allowed for the generation of human-like narrative, where fine tuning of the language model was performed by transfer learning.
- The proposed GPT2 based CBT was demonstrated with a lab prototype implementation.
- The feasibility of generating short sentences by AI that resemble human-generated was experimentally demonstrated. The video capture the live demonstration. For reproducible results, the code is shared online.
- Fine-tunning a pre-trained GPT2 on a synthetic dataset composed of around 5000 short-sentences generated language narratives that help the person look at a situation from a more positive mental outlook.
- The source code is contributed in open source code. This can enable future work by research communities to avert the forthcoming crisis.
The idea of applying GPT-2 for mental health is rather unique in the research literature. GPT2 based CBT is attempted in this work in the backdrop of lack of any research publications on GPT2 based CBT. Given backdrop of a call for multi-disciplinary priority [2] to avert the looming global mental health crisis, contributing to this gap urgently is even more significant. This paper not only address this gap, but also encourages research communities to avert the crisis by 2 contributions as specified in Table 2.
The proposed idea of GPT2 based CBT is introduced to address the challenge articulated earlier. The novel concept of GPT2 in CBT is articulated in Figure 2. In the proposed model, a language model listens to users situation and help her frame the narrative. A language model is simply an AI that predicts the next word in the sentence, given the previous set of words in the sentence. The example illustrated in Figure 2 showed how GPT2 helped a person who lost a job to develop a better ‘outlook’ to pervice the situation. The video demonstration in URL (https://sites.google.com/view/ai-in-mental-health) shows clearly various scenarios of how GPT2 based NLG can help ‘tune’ an individual’s view or outlook. A fine tuned GPT2 model is demonstrated to generate human like text in this video as well in the screenshots in Figure 2 and Figure 4. This demonstrated the novel concept of employing a fine tuned GPT2 model towards a solution for mental health therapy. This demonstration of a novel application of the power of GPT2 to provide human like text to enable therapy will be of tremendous interest to experts and leaders who are interested to prevent the forthcoming mental health crisis caused by the COVID-19 pandemic.
Further, this paper conceptualizes and implements a working prototype of AI based solution that can offer immediate mental health “care” to improve mental wellness of millions of individuals. A working prototype of applying GPT2 for mental health is demonstrated. Also, a video recording of the prototype is presented as a video exhibit in the URL, https://sites.google.com/view/ai-in-mental-health. Further to enable reproducibility of results, the Google colab hyperlinks are shared at https://sites.google.com/view/ai-in-mental-health/ai-to-seed-good-thoughts. The Deep Learning implementation aspects of this Result #1 are further detailed in Result #3.
Given the vast untapped potential of applying Transformers based mental health therapy solutions, this paper also contributes an AI prototype in open source. Thus this paper encourages and enables research communities to accelerate future research to avert the impending & monumental mental health crisis.
Different aspects of the idea are articulated in 4 different pictures - Figure 2, Figure 3, Figure 4 and Figure 6.
The conceptual idea of AI in Cognitive Behavior Therapy is proposed in Figure 2. As illustrated in the figure, the cognitive circle of thoughts and beliefs of an individual is intercepted by self-help based AI therapy. The diagram depicts how AI influence feelings & thoughts. The triangle at the center of the circle represents the person’s beliefs, which can be influenced by the AI. Inspired by CBT technique [17] to get help by correcting one’s beliefs about a situation, the AI offers self-help to correct one’s belief for every situation. An example of a situation is a person losing a job due to the pandemic, and then getting depressed. Figure 2 illustrates an instance of how AI can help a person who may feel depressed after losing his job. In the illustrated scenario, he types/speaks his situation in his smartphone as “I lost my job, I am depressed”. The GPT2 language model takes this initial phrase as an input and predicts the next words in the sentence. Based on this input, the GPT2 generates a narrative as “I lost my job, I am depressed. Let me keep remembering that I am smart”. The screenshot in figure 2 demonstrates this scenario. More examples that demonstrate the AI based self-help is shown in Table 4. A video demonstration of this AI based self-help therapy can be seen online at the URL. The conditional language generation was fine-tuned in such a way so that the positive beliefs are gradually sowed. Mental health experts compose narratives such as one generated in Table 4 containing a situation and a belief into a training dataset. The training dataset used in this prototype had around 5000 such short-sentences containing various situations. Each sentence in the dataset had a situation and the corresponding belief that can help the person to come out of depression. The dataset used in this lab prototype can be accessed at this URL. This dataset is a small dataset synthesized programmatically. Mental Health Professionals compile such a dataset and use it to train the AI to create a fine-tuned GPT2 model. This helps scale the number of patients who can be cared for by every mental health expert. Thus mental health experts can leverage AI to multiply their productivity to achieve the broader objective of preventing the forthcoming mental health crisis across countries. This addresses the shortage of capacity in mental health professionals [31] to scale to millions of families. Mental health professionals fine tune a pre-trained GPT2 model to create a new model using transfer learning. Transfer learning [13, 22] was performed on an OpenAI’s pre-trained GPT2 model [25] as shown in Figure 3. This allows for human like text narratives [21] to be generated by the new model, which when read, may influence the thoughts, hence enabling the depressed individual to cultivate a positive mental outlook [8]. This Human-Computer Interaction model [33] is proposed to be similar to a user downloadable smartphone keyboard such as the popularly used predictive keyboard on android smartphones such as Google Gboard [14]. A visual of the keyboard is shown in Figure 4. The AI model is embedded in the smartphone keyboard, where the AI inference happens locally on the local smartphone device running Tensorflow Lite. So similar to how predictive keyboard such as Gboard helps auto-correct the spelling of what is being typed, the proposed keyboard helps correct the mental outlook to improve the mental health of the smartphone user. Once a smartphone is enabled with this AI, it equips the individual to think more positively, as shown in Figure 2. User’s thoughts, often in the form of speech or text is fed into their personal smartphone, then analyzed in privacy safe technique. Privacy is enabled as the user’s thoughts/text doesn’t leave the smartphone, but inferred locally on the device as presented in Figure 6. Thus a novel conceptual advance in the application of the latest techniques in AI for averting the global mental health crisis had been contributed. Additionally, the proposed AI design is implemented and demonstrated to be technically feasible using a working implementation of the proposed AI. For reproducibility of results, the AI is contributed in open source at the website accessible at this URL. In addition, the additional novel ideas are contributed and demonstrated to evolve an AI solution to address the global mental health crisis,
If the AI identifies a trajectory towards depression, conditional language modelling (NLG) is activated, as shown in Figure 6. A NLU model activates the NLG, based on detection of mental resilience of the individual, as illustrated in Figure 6. At an appropriate instance, a GPT2 neural network-based NLG (Natural Language Generation) transforms any depression thoughts into something with a better outlook. A method to detect mental resilience is discussed later as part of Result 2. Given the set of words as input, GPT2 outputs the next set of words to auto-complete the sentence. So this AI can be used to lead a stream of thoughts away from depression. This novel idea of AI in Cognitive Behavior Therapy is proposed in Figure 2 . AI was demonstrated to generate a narrative that helps influence inner belief for a situation. Also, the proposed AI architecture is presented in Figure 6.
The online demo of AI-based prototype solution is in this URL. In this video exhibit in this URL, it can be seen how a fine tuned GPT2 model can be employed by communities to help the individual to gain a better outlook by fixing the internal beliefs. The fine tuned GPT2 model is able to generate a language of gratitude and hope, even when the input is thoughts of loss and depression, as seen in the screenshot of Figure 4.
Mental health assistance in real-time, where self-correction is facilitated by AI using NLG, while another NLU model keeps a tab on the person’s mental health, was demonstrated in Figure 4 and Figure 6. The comprehensive NLU activated NLG implementation method is discussed in Result 3. The NLU based mental health detection is discussed later as part of Result 2. In short, this result #1 demonstrated the potential and feasibility of applying GPT2 to offer mental health care almost instantly to a potential candidate. This kind of early intervention in mental healthcare is much needed [44].
To summarize, mental health experts train a GPT2 model to multiply their impact to avert the crisis. The novel concept of AI in CBT shown in Figure 2 was demonstrated with a prototype implementation. The potential for proposed AI architecture to solve the the 3 dimensions of the challenge is a notable discussion point. The proposed AI architecture approach presented in Figure 6 is designed to achieve the scalability to millions, achieve early intervention in mental healthcare, and ensure the sensitive personal information of the individual doesn’t leave her personal device for privacy safety. Thus this result is a leap forward in the roadmap to apply recent advances in AI to avert the looming pandemic scale mental health crisis. The open-sourcing of the AI further encourages many research communities.
2.2. Result #2: Implementation of a lab prototype of a NLU based detection of the state of mental health
Challenges addressed by this result:
- Many reports have already established the monumental scale of the mental health challenge in COVID [2]. Given the vast majority of the population across multiple countries, a systematic and scalable strategy for proactive mental health screening and non-intrusive rapid diagnosis is necessary to avert the looming mental health crisis. Figure 1 shows the framework of a solution.
- An idea of NLU based activation of therapy is later discussed as part of the next result, result #3. This implementation of a NLU to detect the progression of mental health is discussed in result #2. This NLU module is later re-purposed as a sub-module as part of result #3.
Methods & Discussions on NLU for detecting mental health:
Very large deep neural networks such as transformer architecture based language models such as Google BERT offer a significant ability to understand English language, making it an excellent choice to understand what a candidate says using NLU (Natural Language Understanding) [12]. Transfer Learning on BERT is proven to be a viable technique for understanding sentences in any domain [13]. The abundance of literature in NLU for social media listening [5, 34] is noted in Table 1. Within the context of social media listening with NLU, the idea of use of BERT (Bidirectional Encoder Representations from Transformers) [10] over tweets is well established recently in 2019. Until 2018, Deep Neural Networks such as RNN was utilized [5]. This result shows the application of these state of art Deep Learning methods for screening or diagnostic. Given an abundance of literature in NLU based screening, this result #2 is about an open-source lab prototype implementation based on proven approaches.
Result & Method:
Highlights of Result #2:
- Result #2 is about the application of NLU to detect the state of mental health of an individual.
- Result #2 assumes significance in the context of the idea of NLU activated NLG, the details of which are described later as part of Result #3.
- Result #2 is about implementing an NLU module to detect the progression of an individual’s mental health.
- The implementation is by application of BERT.
- From a literature point of view, the paper doesn’t claim any novelty in result #2 on its own. Result #2 is presented here for two reasons. Firstly, it is re-purposed as a sub-module in Result #3. Secondly, it is part of a contribution to avert the crisis as specified in Table 2.
To address the scale of a pandemic, a very different approach is necessary. Any capability to perform large scale screening or rapid pre-diagnosis enable experts is valuable to avert the looming crisis. A way to quickly analyze what a candidate is gone through during past weeks is the result obtained. An ability to screen large number of public, and analyze temporal patterns of every candidate quickly in the form of a visual report – makes it possible for rapid diagnosis by mental health professionals. An AI solution for large-scale screening is shown in Figure 7 to identify candidates who need diagnosis. For the shortlisted candidates, a visual report, as shown in Figure 5 is generated for each candidate. The visual reports show mental resilience and time-based swings of cognitive behavior and the recovery rate. This report enables rapid diagnosis by mental health therapists. Using transfer learning, a BERT [10] based binary classifier identifies if the person is showing the language of a depressed person or exhibiting the signs of a person on a recovery path. The time to recover after a loss of job or family member is also explored. The mental resilience of a candidate can be understood by seeing the trend over a period of time. The proposed architecture for the detection of progression of mental health is shown in Figure 5. In this figure, a topic analyzer is cascaded with a mental health classifier, and then temporal modeling is performed. In short, a quick way to pre-diagnose by mental health professionals is demonstrated by the application of state of the art NLU. Thus the feasibility of screening at scale (Figure 7) along with rapid pre-diagnosis by mental health professionals using a cascade of BERTs based architecture (Figure 5) is demonstrated. An online demo is at this URL, https://sites.google.com/view/ai-in-mental-health/ai-in-diagnosis.
2.3 Result #3: Design of neural network architecture: NLG activated GPT2 based therapy
Challenges addressed by this result:
The challenge is in coming up with a neural network architecture for the solution proposed earlier in Result #1. The challenges to be addressed are
- Scalability to a vast population
- Privacy & protecting sensitive information: Patients reveal a lot of personal experience during a mental health counseling session. Collection of such sensitive data from many patients by a cloud system may involve risks. Such data should NOT leave the personal device boundary and should be deleted immediately after AI processing on the personal device.
- Early intervention and 24x7 availability of care
- Activate assistance only when appropriate: Intelligently choosing when to assist the patient in contrast to enabling her to become self-reliant.
Methods & Discussions:
The opportunity to scale impact via AI based CBT is discussed earlier in Result #1. Here familiar technique of transfer learning is used on the dataset on OpenAI’s pre-trained GPT2 language model. This transfer learning idea is shown in Figure 8. Transfer learning in GPT2 has successfully produced poetry [15] and fake news [21]. The superiority of GPT2 (Generative Pretrained Transformer 2) [25] over earlier NLG techniques to produce reasonable text generation has been well established due to attention [18] based neural network architecture on model capacity above 100 million parameters.
The therapy needs to be started only when appropriate. If the person shows signs of natural recovery from a depressed mental state due to natural resilience, therapy is NOT required. So the AI therapy needs to be activated only when appropriate. It is important to keep a tab on the person’s progression of mental health. Based on the progression, the decision to activate is taken intelligently by the AI. This idea of intelligent activation of therapy based on how the person is doing over few days is unique in the literature. This concept of detection by NLU, and then appropriate activation of therapy is proposed and demonstrated in this paper. The proposed intelligent activation approach is shown in Figure 6. A BERT based model keeps track of a person’s mental health. In case this model detects that a person is trending towards depression, then it triggers activation of GPT2 based therapy model. This idea of intelligent activation of therapy using NLU activated NLG architecture is unique in the literature of mental health. The result is demonstrated in Figure 11.
Result & Method:
Highlights of Result #3:
- The idea of NLU activated NLG based therapy was proposed and implemented. A BERT-GPT2 cascade was implemented. In this approach, BERT detects if a person is depressed, and when appropriate, activates the AI therapy. The results of the proposed neural network architecture design presented in Figure 8 is demonstrated in Figure 11. Figure 11 shows two scenarios from a prototype of NLU activated NLG therapy. One scenario showed NLU triggered GPT2 therapy, another where therapy was NOT required. This demonstrated the proposed AI architecture of NLU triggered NLG. This intelligent activation architecture was demonstrated. This result is useful for early intervention [44], where the AI is pre-deployed in smartphones and gets automatically activated at the right time for timely activation of therapy to enable early intervention.
- For reproducible results, the source code is accessible online at URL.
- A pre-trained GPT2 model is fine tuned using transfer learning on a small synthetic dataset. Three different pre-trained GPT2 models of small (124M), medium (355M) and large (774M) pre-trained models were fine-tuned and all there achieved the same accuracy levels. (Refer Figure 9)
- Since the small (124M) and large GPT2 (774M) achieved the same accuracy levels for a small dataset of 5000 short sentences, a small GPT2 model with 124 million neural network parameters will be appropriate for on-device GPT2 inference.
- Though GPT-3 is a successor to GPT-2, GPT-3 is not suitable for on-device inferenc This paper experimentally identified a small GPT2 model is sufficient to provide the performance required. In the future, researchers can explore distillation/model compaction and smaller model such as DistillGPT2 [36]
- Human-like text generation capability of NLG activated GPT2 based therapy was demonstrated with reproducible results. The sample results in Table 4 show human-like text generation is feasible with Beam search on GPT2. This demonstration of the capability to generate human like short narratives is useful as AI can offer early CBT therapy in the form of timely self-help, given mental health professionals need tools to multiply their productivity.
- This result demonstrated AI based self-help therapy. Its advantage is early intervention in mental health, user privacy & massive scalability. Thus, an AI-based mental health strategy & neural network architecture that combines the strengths of latest advances in NLU and NLG, namely BERT and GPT-2 was proposed and demonstrated. Thus this result demonstrated a way forward in preventing of looming pandemic scale mental health crisis.
- The open source contribution encourages many communities for further research, given experts are calling to action for multi-disciplinary research.
To summarize, here are unique knowledge elements and results contributed
- Beam search [26] with GPT-2 decoded the next words in the working prototype. The results of beam search based GPT2 prediction is shown in the screenshot in Figure 4. The prototypes are implemented with Transformers [28, 29] based NLG and NLU using a Tensorflow/Keras library [29] on Google colab.
- A novel NLU-triggered-GPT2 is proposed & demonstrated with a prototype implementation. The proposed architecture of NLU triggered NLG to selectively activate the therapy is presented in Figure 6 and more detailed in Figure 8. The simple implementation of the proposed architecture as a lab prototype is demonstrated online at the URL, and the screenshot of this is presented in Figure 11. The screenshots in Figure 11 shows the two different scenarios, one where the AI therapy is activated, another scenario where the therapy was NOT required. The activation of GPT2 is performed only when appropriate as NLU module keeps a tab on the person’s mental health over time. By turning on/off the AI on the smartphone keyboard, families can opt in for self-help based mental wellness. So when both mental health tracking AI, and CBT AI are combined together, this allows a pervasive, non-intrusive, privacy safe approach to provide mental health care for millions. The BERT model detects if a person is depressed, then selectively activates the GPT2 based therapy. NLU module's ability to detect the progression of mental health over a series of sentences over days was shown earlier in result #2. Figure 5 earlier showed that the NLU was able to detect if a person was recovering from depression. Hence the idea of NLU triggered NLG allows for activation of therapy only during appropriate circumstances such as the person doesn’t recover after a loss. While most people recover from a loss naturally after a duration, some may get into increasing levels of depression. This proposed idea of NLU activated therapy is able to handle such situations who need help. The proposed HCI model of smartphone keyboard makes it easy to keep a tab of the mental health of a family member, and the proposed approach can automatically activate therapy in the form of self-help. While users can opt-in to automatically download ‘mental wellness AI’ feature in their smartphone keyboard, this ensures family members stay mentally healthy. The protection of user’s sensitive information was demonstrated earlier in Result #1. Hence the path for scalable timely & early intervention with proactive deployment of the proposed ‘NLU activated NLG’ for masses can protect a country from being impacted by any future mental health crisis.
- Three pre-trained GPT2 model of different sizes were fine tuned with same synthetic dataset. These were small , medium and large GPT2 neural network models of 174, 355, 744 million parameters The performance results of 3 different GPT2 models are tabulated in Table 3 and plotted in Figure 9 as training loss over training steps. This showed that for a small synthetic dataset, all 3 models achieved the same level of performance. Given the on-device requirement for AI inference, the GPT2 small (174M) model is suggested for smartphone deployment. In future, researchers can explore more memory and compute efficient models for smartphone such as DistillGPT2 [36] to enable deployment into real world.
- An idea of conditioning language generation is demonstrated, demonstrating that AI is close to generating narratives like a mental health professional. The resulting conditioned text is shown in Table 4 for a couple of scenarios. It shows how AI can assist during many situations. It shows AI generates a conditional narrative that spins the words so that the user can look at the situation from a positive mental outlook. Transfer learning on Transformers based language model opens up the “imagenet moment for NLG”, so the potential of GPT2 for mental health therapy was successfully demonstrated. This result is a significance step towards AI based mental healthcare. The potential of Transfer learning in GPT2 to transforming the boundaries of what is feasible.
- The dataset used for transfer learning for training the GPT2 model is a programmatically generated synthetic dataset. This synthetic dataset used in this paper contains 4098 records and can be accessed at the URL, https://sites.google.com/view/ai-in-mental-health/ai-to-seed-good-thoughts. Given the dataset is programmatically created as per the code shared at this URL, this gives flexibility for the Mental health therapists to quickly configure the input words to generate a synthetic dataset for a many situations. The dataset is contains short sentences such as the one shown in Table 4. Each sentence have a combination of a situation, and a belief. The situation and belief to influence is created by a mental health experts. The training time for all 3 models shown in Table 3 on this dataset was completed in less than 5 minutes on a single GPU environment for all the different sized GPT2 models, enabling swift solution deployment by mental health experts at the time of real word deployment.
- Since mental health counselling for a set of target population related to each other by similar situations (e.g. a group of nurses overwhelmed by handling COVID-19 patients in a hospital), federated learning approach can be beneficial. To learn from community, aggregate model of Federated Learning [23] can be explored. The federation learning concept is presented in the architecture in Figure 8. Federated learning to identify positive self-help narratives that yield faster healing based on joint learning from a group of similar patients. This could be a future direction of research.
- The choice of AI inference on the smartphone vs cloud was discussed in the context of privacy and willingness of patients to send sensitive information to cloud. A smartphone based model with AI inference on the local device enables privacy safety and user willing-ness to express themselves. The architecture proposed in Figure 6 ensures the user’s personal thoughts doesn’t leave her personal smartphone device, and hence enables privacy.
Based on the results demonstrated here, future researchers can develop a practical real world mental health solution to avert the forthcoming mental health crisis.
With the intelligent activation of AI therapy using a BERT-triggered-GPT2 architecture presented in Figure 6, Figure 11 shows the potential of AI based mental health solution to address the monumental challenge the paper aimed to achieve.
The power of transfer learning capable Natural Language Generation models such GPT2 when combined with on-device deployment approaches as shown in Figure 8 represent a leap forward in solution to respond to the global call for action by WHO and experts.
The design of neural network such as one in Figure 8 with an Open source lab prototype implementation demonstrated is a step forward in the response to the call for action by WHO to prevent the forthcoming monumental crisis.
The source code is available at URL https://sites.google.com/view/ai-in-mental-health/ai-to-seed-good-thoughts. The AI prototype is open-sourced. This encourages research communities for future research, given experts have called for multi-disciplinary priorities to avert the crisis.