Overview
A three-step approach to identify the priority outcome domains will be used. First, a draft comprehensive list of outcomes will be compiled through a review of outcomes reported in Cochrane reviews and existing randomized controlled trials, focus group interviews with key stakeholders, and online surveys of service users and caregivers. Second, the comprehensive outcome list will be refined through a pilot study. Third, a Delphi study will be conducted using the comprehensive outcome list to develop outcome lists (Fig. 1). Our research team includes researchers with backgrounds in academic medicine, nursing, social work, occupational therapy, music therapy, clinical psychology, and service user experience. The multidisciplinary research team has conducted or will conduct all the stages of this study and report the findings. In other words, all the process of the study has involved or will involve user-researchers.
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. This study protocol was approved by the Tokyo Metropolitan University Ethics Committee (No. 20083) and the Research Ethics Committee of the National Center of Neurology and Psychiatry (No. A2021-005). The study protocol was registered in the University Hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN-CTR ID, UMIN000044680).
Step 1: Development of a comprehensive list of outcomes that are considered important by service users, caregivers, and stakeholders in Japanese community mental health research
To inform the design of the Delphi surveys, the sections on Steps 1 and 2 describe what we have already done. However, no substantive findings are presented in this paper. During Step 1, we developed a long list of outcomes using the results from focus group interviews and online questionnaires, in addition to the data collected through a literature review.
1–1. Literature review
A review of literature was conducted to ensure identification of an exhaustive list of relevant outcome domains. We took this approach with the goal of comprehensively examining the outcomes that researchers have used in past studies. Details of the literature review can be found in an article by Igarashi et al (Igarashi et al, in submission). A literature search was performed using electronic databases (Cochrane Library, MEDLINE, PsycINFO, and Ichushi-Web) for any original research published in English or Japanese. We searched for articles using a combination of keyword searches related to community mental health care and severe mental illness such as "severe/serious mental illness" and "community mental health" and all articles for the leading to the search date were reviewed. Papers were therefore excluded first on the basis of their titles and then on the basis of their abstracts. Details of the inclusion and exclusion criteria are shown in Supplementary Table 1. Title and abstract screening and full-text screening were independently completed by at least two authors for Cochrane Library (MI, TK, and SY), MEDLINE, PsycINFO (TK and SY), and Ichushi-Web (TS and SY). Any discrepancies were resolved through discussion with a co-author (CF). For each article, we extracted all outcomes specified in the Methods section. More specifically, we identified outcomes that indicated what factor was measured rather than descriptions of how it was measured. We classified the outcomes identified from each article and added them to the long list of outcomes.
1–2. Focus group interviews
In September 2019, group interviews were conducted with service users and other stakeholders. The first half of the group interviews were conducted with the same stakeholders and the second half of the interviews were conducted with a mixture of stakeholders. Interviews were conducted to examine outcomes that stakeholders, including service users and caregivers, thought were important. The details of this research are described in an article by Ogawa et al (Ogawa et al, in preparation).
Participants and recruitment
Service users, caregivers, community mental health service professionals (e.g., doctors, nurses, psychiatric social workers, occupational therapists, clinical psychologists), national or regional government staff that work in mental health, and researchers who have studied community mental health participated in the group interviews. Inclusion criteria were (1) ability to respond to the interview content voluntarily, (2) living in Japan, and (3) native Japanese speaker. Exclusion criteria were (1) current hospitalization, (2) age under 20 years, and (3) adult guardianship. We recruited participants by explaining the purpose and content of this study to service users or caregiver organizations, professional associations, and academic societies. Regarding participants with a research background, we recruited authors of original papers on community psychiatry and community mental health services in addition to recruiting members of relevant academic societies. The following participants joined the group interviews: 10 persons with mental illness, 6 caregivers, 10 community mental health service workers, 4 government staff, and 11 community mental health researchers. We explained the purpose of the study and the details of the research methodology to the participants. Written informed consent was obtained from all participants.
Methods
In the group interviews, participants were divided into groups according to each attribute and discussed the theme of "What are the outcomes that should be measured in research on community psychiatry and community mental health services?" One research team member participated in each group as a facilitator to encourage and organize input. At the end of the group interviews, the outcome results from each group were shared with all groups. The content of the interviews were audio recorded and transcribed. In addition, basic information on the participants such as age and gender collected before the interview was also tabulated.
Analysis of the focus group interviews
We transcribed the recorded interviews. We collected statements that we judged to be about "outcomes that should be measured in research on community psychiatry and community mental health services." The validity of the text for analysis was ensured by having the research team members (SY, MO, NY, and MA) work in pairs to confirm the text and adopt the parts that one or more of pair judged to be relevant for analysis. We summarized the extracted statements and coded them as outcome items. The coded outcome items were categorized and combined with similar items by research team members (MO, NY, and TS).
1–3. Online questionnaire surveys
Study design
In order to collect a wide range of potential outcomes from the perspective of service users and caregivers, we conducted two online surveys. One was an online cross-sectional survey of caregivers (Yasuma et al, in submission). The other survey was an online survey of service users who lived in the community (Shiozawa et al, in preparation).
Participants and recruitment
(1) Caregivers
Participants were recruited from the LINE Family Association of Schizophrenia. This family association had 238 registered members as of August 1, 2020. It offered a platform for information exchange and peer support for caregivers of people with schizophrenia using the LINE online software. Eligibility criteria were as follows: (a) caregiver of a person with schizophrenia and (b) age over 20 years. For example, a caregiver can be someone who cares for a family member with schizophrenia and looks after his or her daily living needs.
(2) Service users
Participants were recruited from the Community Mental Health & Welfare Bonding Organization (COMHBO). The COMHBO had 500 registered members who are "Combo Writer" service users that introduce their experiences in an institutional journal. Eligibility criteria were as follows: (a) ability to respond to the interview content voluntarily and (b) age over 20 years. Exclusion criteria were (a) current hospitalization and (b) adult guardianship.
Methods
The explanatory text of this survey and the URL of the website for the questionnaire survey were sent to potential participants via the mailing list of their organizations. Potential participants were asked to access the website. Potential participants read the details of the explanatory text about the study at the beginning of the web page and continued to answer only if they agreed to participate in the study. In addition to the basic information of participants, we set an open-ended question to capture the important outcomes for service users or caregivers of people with schizophrenia in community mental health research. The question was worded as: “What do you want to become in the future after receiving support? Please enter up to three of your own ideas.”
Analysis of the online questionnaire surveys
Three authors (NY, TK, and TS) independently extracted the participants’ responses for outcomes. They adjusted for word fluctuations in the responses. Next, similar responses were collected and categorized through joint analysis and thorough discussion by research team members (NY, TK, and TS). During the categorization process, outcomes with ultimately unclear meanings were excluded.
1–4. Development of the long list of outcomes
The long list of outcome domains was generated by combining the results from the literature review, focus group interviews, and online questionnaire surveys. We extracted 853 outcome items from the literature review, 81 outcome items from the focus group interviews, and 91 items from the two online surveys.
To form an accessible Delphi survey, the outcomes needed to be grouped together where there were areas of commonality or duplication and mapped into outcome domains. Four research meetings were held to remove duplication of outcome items, further consolidate common outcome items, and map outcome domains. Participants in these meetings were from a range of health and social care research backgrounds (doctors, nurses, psychiatric social workers, occupational therapists, clinical psychologists, experts-by-experience). In addition, research team members (MI and TS) held multiple meetings to discuss and organize outcome items. All outcome items collected were maintained on an Excel spreadsheet. Research team members laid out the outcome items printed on paper on a table and summarized similar outcome items or deleted outcome items that were determined to be inconsistent with the purpose of this study. Any disagreements were resolved through discussion. Finally, 94 outcome items were identified (e.g., Psychiatric symptoms or mental state; Physical health; Activity or leisure; Employment, life satisfaction, quality of life, or well-being) and included in the Delphi study (Table 1). We defined each outcome item and created a summary to understand the meaning of the items.
Table 1
Long list of outcome domains
1.
|
Death - suicide
|
41.
|
Religion or beliefs
|
2.
|
Death - all causes
|
42.
|
Sexual satisfaction
|
3.
|
Psychiatric symptoms or mental state
|
43.
|
Cognitive functioning
|
4.
|
Psychological distress (anxiety or depression)
|
44.
|
Knowledge of illness and services
|
5.
|
Relapse or remission
|
45.
|
Medication adherence
|
6.
|
Insight
|
46.
|
Treatment adherence
|
7.
|
Substance use
|
47.
|
Attitudes towards medication or treatment
|
8.
|
Self-harm
|
48.
|
Satisfaction with services
|
9.
|
Violence or aggression
|
49.
|
Unmet needs
|
10.
|
Suicidal ideation or attempt
|
50.
|
Perceived coercion
|
11.
|
Laboratory measures
|
51.
|
Housing stability
|
12.
|
Physical health
|
52.
|
Earnings
|
13.
|
Weight and obesity
|
53.
|
Duration of community living
|
14.
|
Physical fitness
|
54.
|
Family relationships or functioning
|
15.
|
Chronic pain
|
55.
|
Living with family
|
16.
|
Self-care
|
56.
|
Having a role model
|
17.
|
Perceived stress
|
57.
|
Therapeutic relationship
|
18.
|
Subjective health status
|
58.
|
Costs of mental health care
|
19.
|
Overall functioning
|
59.
|
Costs of all care
|
20.
|
Daily living skills
|
60.
|
Mental health service use
|
21.
|
Contact with the legal system
|
61.
|
Hospital admission
|
22.
|
Interpersonal relations
|
62.
|
Involuntary treatment
|
23.
|
Help-seeking
|
63.
|
Duration of admission
|
24.
|
Partner or marriage
|
64.
|
Outpatient visits
|
25.
|
Overall social functioning
|
65.
|
Emergency service use
|
26.
|
Independent living
|
66.
|
Non-psychiatric service use
|
27.
|
Social connectedness
|
67.
|
Number of caregivers needed to maintain stable state
|
28.
|
Activities or leisure
|
68.
|
Medication prescription
|
29.
|
Place of safety and belonging
|
69.
|
Adverse events or effects
|
30.
|
Employment
|
70.
|
Caregivers' mental health
|
31.
|
Work-related skills or vocational ability
|
71.
|
Caregivers' physical health
|
32.
|
Absence from or leaving a job
|
72.
|
Caregivers' subjective health status
|
33.
|
Job hunting and related activities
|
73.
|
Family's stigma and discrimination
|
34.
|
Duration of employment or hours worked
|
74.
|
Caregivers' problem-solving or coping skills
|
35.
|
Childbirth and childcare
|
75.
|
Caregivers' self-esteem
|
36.
|
Caregiving for family members
|
76.
|
Caregivers' knowledge of illness and services
|
37.
|
Education
|
77.
|
Caregivers' service satisfaction
|
38.
|
Role in society
|
78.
|
Caregivers' life satisfaction or quality of life
|
39.
|
Peer support
|
79.
|
Expressed emotions
|
40.
|
Life satisfaction, quality of life, or well-being
|
80.
|
Caregivers' service use
|
81.
|
Motivation
|
88.
|
Social support for caregivers
|
82.
|
Empowerment or self-determination
|
89.
|
Experience of caregiving
|
83.
|
Self-esteem
|
90.
|
Burden of care
|
84.
|
Resilience
|
91.
|
Financial burden of care
|
85.
|
Feeling dependent on psychiatric treatments
|
92.
|
Influence on caregivers' lifestyles
|
86.
|
Symptom control skills or coping
|
93.
|
Influence on caregivers' jobs
|
87.
|
Stigma and discrimination
|
94.
|
Influence on caregivers' leisure activities
|
Step 2: Pilot study of the outcome list
We will conduct a pilot study for the purpose of determining the adequacy of the long list and descriptions for each outcome item developed in Step 1. The pilot study also aims to obtain feedback on wording, usability, and visuality for the online survey form.
Participants and recruitment
Participants in the pilot study will be service users, caregivers, community mental health service providers, government staff, and community mental health researchers. We will recruit two participants from each group.
Data collection and analysis
In the pilot study, two rounds of online questionnaire surveys will be conducted using a website.
<Round one>
(1) We will send the participants an invitation to the survey website by e-mail. We will ask them to respond to the survey from the dedicated webpage. Potential participants will read the explanation of the study at the beginning of the webpage and then will answer only if they agree to participate in the study.
(2) The long list of outcomes generated during the Step 1 will be presented to participants. Participants will rate each outcome item in the long list on a seven-point scale from "very important" to "not important at all." In addition, participants can write comments if they have opinions on each outcome item. They can also suggest new outcome items. Furthermore, the usability of the survey website and the clarity of the descriptions will be evaluated.
(3) We will compile and summarize the participants’ comments. Results will be presented as averages, percentage of respondents with each score, and variance so that participants can confirm the variation in others’ opinions. In order to avoid the risk of adjustment bias by our research team, all comments will be reflected in the summary. Corrections to wording and additions of items to the long item list will be made after discussion among research team members.
<Round two>
(1) We will report the results of the first round to the participants. After confirming the results of the first round, participants will respond to the revised long list of outcomes.
(2) Participants will rate each outcome in the revised long list on a seven-point scale from "very important" to "not important at all." Participants will enter and submit free-text descriptions for each outcome.
Using the results of the pilot survey responses, we will draft the "Long list of outcomes for community mental health research in Japan" to be used in the Delphi survey.
Step 3: Delphi study
An online Delphi survey will be conducted to develop the priority outcome list for the community mental health setting in Japan. Since there are no definitive or optimal methods for conducting a Delphi survey, we will determine the survey methods accordingly with reference to previous studies.
1) Number of study participants
The Delphi survey for five panel attributes will include service users, caregivers, community mental health service providers, national or regional government staff that work in mental health, and community mental health researchers. While there is no clear criterion for the number of participants in a Delphi study, studies usually include 50 members per panel, according to a relevant review [16]. With reference to previous studies, the number of expected participants in this study will be 250, consisting of 50 people for each panel attribute.
2) Response rate
In general, a response rate over 70% is required to obtain reliable results in Delphi studies [17]. This study will follow previous studies and aim for a response rate of 70% or higher for each round.
3) Number of iterations for consensus building
Given the risk of participant burden and thus a lower response rate, a Delphi study in the field of health services generally needs two to three rounds for consensus building [18, 19]. Based on previous studies, this study will include a total of three rounds.
4). Criteria for consensus
The definition of consensus in a prior study was 70% or more of respondents rating an outcome item as "important" or "very important” [17]. In a review article, the basic definition was set in the range of 55–100%, and since 70% was reported the most of using as criterion [20]. The definition of consensus for this study is 70% or more. In addition, we will exclude items if 75% or more of the respondents rated them as “not important” or “not important at all” based on a past core outcome study of discharge interventions among service users [21].
Participants and recruitment
The potential participants of the Delphi survey will be stakeholders in community mental health care, as in the pilot surveys. There will be 50 stakeholders of each type. Considering a 11.7% attrition rate (about 6 participants) based on a previous study [22], we will recruit approximately 60 participants in each stakeholder group. Inclusion criteria include (1) ability to respond to the study voluntarily and (2) living in Japan. Exclusion criteria include (1) currently hospitalization, (2) age under 20 years, and (3) adult guardianship.
Other inclusion criteria will be established for each attribute panel in this study. The inclusion criteria for service users are: (1) diagnosis of mental illness (potential participants with dementia and intellectual disability are excluded from participation), (2) disability that affects daily life (serious mental illness), and (3) use of community mental health and social services. The inclusion criteria for caregivers are: (1) primary caregiver for a service user. The inclusion criteria for community mental health service providers are: (1) national license in medical health and welfare that have (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist, and other) and (2) history of involvement in community mental health and welfare services. This study will recruit at least five people from each major mental health profession (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist) in order to reduce the occupational bias of participating supporters. The inclusion criteria for national or regional government staff are: (1) working for a government agency and (2) involvement in work related to community mental health and social services. The inclusion criteria for researchers are: (1) doing research in the field of community mental health care. We also asked whether the researcher held a national license in medical health and welfare. As in the community mental health service providers, we will recruit at least five people from each mental health profession (doctor, nurse, psychiatric social worker, occupational therapist, clinical psychologist, and other) to participate in the study.
We will recruit participants by explaining the purpose and content of this study to services user or caregiver organizations, professional associations, and academic societies. We will explain the survey outline to key persons in each organization. If the organizations or key persons agree to collaborate on the Delphi survey, they will inform their fellow members about this survey. We will also ask the participate to focus group interview participants and authors of the original papers on community psychiatry and community mental health services.
Delphi Round 1
In the first round, participants will be asked to register online. Each participant will be asked to identify the applicable stakeholder group. A unique identifier will be assigned to each participant to allow identification of individuals that complete each round. Potential participants who are informed about the Delphi study via announcements from associations or emails from the research team will visit a webpage introducing the Delphi study web page of Department of Community Mental Health & Law (URL:https://www.ncnp.go.jp/nimh/chiiki/research/32.html). If an individual is interested in participating in the Delphi study, they will provide an email address and obtain an invitation URL for the survey website. As in the pilot study, participants will read the detailed explanation of the study at the beginning of the webpage. They indicate agreement to study participation by participating in the survey. The long list identified through Steps 1 and 2 will be presented to participants. Participants will rate each outcome item on a seven-point scale from "very important" to "not important at all." In addition, participants can submit comments for each outcome item. They can also suggest new outcome items if they think that there are essential outcome items that were not in the long outcome list.
Round 1 data analysis
The response rate will be assessed at the end of Round 1. The total number of respondents completing the round will be compared to the number of respondents who agreed to participate in this study. Response rate will be calculated by the number of participants overall and by stakeholder group (service user, caregiver, community mental health service providers, government staff, and researcher). For each outcome item, the distribution of scores will be summarized and analyzed. For each outcome item in Round 1, the proportion of participants scoring 1–2, 3–5, and 6–7 on the seven-point Likert scale will be calculated for each item. In addition, the results will be presented using averages, percentages of respondents with each score, and variance so that variations in opinions can be confirmed. Each outcome will be classified as “consensus in,” “consensus out,” or “no consensus.” “Consensus in,” referring to consensus that the outcome should be included in a core outcome set, will be defined as greater than 70% of participants scoring 6–7 and less than 25% of participants scoring 1–2. “Consensus out,” referring to consensus that the outcome item is not appropriate for include in the priority outcome domain, will be defined as greater than 75% of participants scoring 1–2 and less than 50% of participants scoring 6–7 in all stakeholder groups. “No consensus” is defined as any other distribution of scores. Only outcomes identified as "no consensus" will be re-evaluated in Round 2.
All free-text comments will be reflected in the summary to avoid the risk of adjustment bias introduced by the research team. Modifications to the wording of items or addition of outcomes based on free-text comments will be made after discussion among the research team members.
Delphi Round 2
We will report the results of Round 1 to the participants. After confirming the Round 1 results, participants will respond to the revised questionnaire. Participants will rate each outcome item on a seven-point scale from "very important" to "not important at all." In addition, if participants have any comments on each outcome item, they could write them in the free-text field.
Round 2 data analysis
The response rate will be assessed at the end of Round 2 using the same procedure as in Round 1. In Round 2, changes in the number of respondents from Round 1 will also be assessed. For each outcome item, the number of respondents and distribution of scores will be summarized and analyzed. For each outcome in Round 2, the proportion of participants scoring 1–2, 3–5, and 6–7 on the seven-point Likert scale will be calculated. In addition, the results will be presented using averages, percentages of respondents with each score, and variance so that the participants can understand the various views of others. Based on the consensus criteria, only those outcomes that are identified as "no consensus" will be re-evaluated in Round 3.
Delphi Round 3
We will report the results of Round 2 to the participants. The participants will confirm the Round 2 results. Next, they will respond to the revised questionnaire. As with Round 2, participants will rate each outcome items on a seven-point scale to indicate their priority regarding the outcome items in the list.
Round 3 data analysis
At the end of Round 3, the response rate and number of respondents will be assessed using the same procedure as in the previous rounds. The number of respondents and distribution of scores will be summarized and analyzed for each outcome item. Round 3 data will be analyzed by stakeholder group. Results on outcome items will be presented using averages, percentages of respondents with each score, and variance. Each outcome will be classified as “consensus in,” “consensus out,” or “no consensus.”
Based on the results of Rounds 1–3 of the Delphi survey, the outcomes categorized as “consensus in” by all stakeholder groups will be on the "Important outcome domain list for service users, caregivers, supporters, and researchers in community mental health research in Japan."