Assessment of Catatonia and Inter-Rater Reliability of Three Instruments: A Descriptive Study in an Acute Mental Health Unit in Urban South Africa


 BackgroundClinical assessment of catatonia includes the use of diagnostic systems, like the Diagnostic and Statistical Manual 5 (DSM-5) and International Classification of Disease Manual 10 (ICD-10), or screening tools such as the Bush Francis Screening Instrument (BFCSI)/Bush Francis Catatonia Rating Scale (BFCRS) and Braunig’s Catatonia Rating Scale. In this study, we describe the inter-rater reliability (IRR), utilizing the BFCSI, BFCRS and DSM-5 to screen for catatonia. MethodData from 10 participants recruited as part of a larger prevalence study were used to determine IRR by five assessors after they were trained in the application of the 14-item BFCSI, 23-item BFCRS and DSM-5 to assess catatonia in new admissions. Krippendorff α was used to compute the IRR, and Spearman’s correlation was used to determine concordance between screening tools. The study was performed at a 35-bed acute mental health unit in Dora Nginza Hospital, Nelson Mandela Bay Metro. Participants were mostly involuntary admissions under the Mental Health Care Act of 2002 and between the ages of 13 and 65 years and older.ResultsThe majority (8, 80%) were 35 years or less and males (8, 80%). At least six (60%) of the ten participants were found to have catatonia based on the BFCSI/BFCRS and DSM-5 criteria.The BFCRS (complete 23-item scale) had the greatest level of inter-rater agreement with an α=0.798, while the DSM-5 had the lowest level of inter-rater agreement with an α=0.565. The highest correlation coefficients were observed between the BFCRS and BFCSI.ConclusionThe BFCSI and BFCRS were found to be sensitive tools for screening for catatonia with high IRR and high correlation coefficients, while the DSM 5 showed some deficiencies in screening for catatonia with low IRR and lowest correlation with the other two tools.


BACKGROUND
Catatonia has a wide range of prevalence in different populations, from less than 10% to just above 60%. [1][2][3] This may be influenced by factors that include the assessment tools used to screen for catatonia and the inter-rater reliability (IRR) among clinicians undertaking the assessment. In this descriptive study, we used the Bush Francis Catatonia Screening Instrument (BFCSI), Bush Francis Catatonia Rating Scale (BFCRS) and Diagnostic and Statistical Manual-5 (DSM-5) as assessment tools for catatonia. 4 5 We analyzed data to determine the IRR among a group of five mental health professionals trained to apply these tools in an inpatient acute mental health setting. Data were collected from September 2020 to February 2021 as part of a larger 12-month study on catatonia in Dora Nginza Hospital, a general regional hospital in Nelson Mandela Bay, South Africa.

Assessment of catatonia
Tools or diagnostic systems such as the DSM-5 or International Classification of Diseases 10 (ICD-10), help to guide the clinical examination when assessing catatonia. 3 4 Validity and IRR of assessment tools are important considerations that may influence the pick-up rate of catatonia, which makes it important to choose a valid assessment tool with acceptable IRR. 5 6 The BFCSI meets these criteria as an acceptable screening tool because it has shown good IRR and has been successfully used at the study site to screen new admissions for catatonia. 5 7 A number of studies have also indicated that the BFCSI is a reliable and valid screening tool for catatonia. 3 5 7-10 Comparing the BFCSI and DSM-5 in the assessment of catatonia-The BFCSI was developed by Bush et al. as a 14-item screening tool for catatonia. To assess the severity of catatonia, its use is complemented by completion of the full 23-item BFCRS if two or more signs of catatonia are present. 5 7 The BFCRS has also been recommended by Sienaert et al. for routine use because of its good reliability, validity, and relative ease of application. 9 In a study by Sarkar et al. on assessment of catatonia and IRR using four different instruments, more cases with catatonia were identified when applying the full BFCRS scale compared to the DSM-5, and IRR was demonstrated to be good (α=0.779). 4 10-12 Similar to this study, the number of the assessors in our study were five, consisting of three professional nurses and two psychiatry residents.

Aims
We aimed to investigate and describe the IRR in the assessment of catatonia at the study site using the DSM-5 and BFCSI/BFCRS to screen for catatonia as part of a larger 12-month study on catatonia.

Objectives
The main research objective was the assessment of the first ten participants suspected of having catatonia by five assessors using the BFCSI/BFCRS and DSM-5 to determine IRR in a prevalence study on catatonia. We present data on the IRR and assessment of catatonia at the study site.

Research design
This was a prospective, descriptive study that utilized a quantitative method. Data based on the BFCSI/BFCRS and DSM-5 criteria were used to calculate the IRR and concordance rates for the assessment tools.

METHODS
Patients who were admitted to the MHU from September 2020 to the February 2021 were requested to participate in the study. Those who provided informed consent were recruited by the research team comprised of the lead researcher and five research assistants. For the IRR evaluation, the first 10 participants suspected of having catatonia on admission as determined by the admitting doctor, were assessed by the research team. The screening tools applied by the research team were the BFCSI/BFCRS and DSM-5 to assess for the presence or absence of catatonia, and the data collected were used to determine the IRR.

Outline of the study process
The research team screened all new admissions for catatonia. Assessors were two psychiatry residents, both with two years of experience in psychiatry, and three mental health professional nurses with a background of more than ten years each, working in mental health services.
The principal investigator trained the five assessors on how to use the BFCSI/BFCRS and DSM-5 to assess for catatonia. The training consisted of the meaning of the terms used in the BFCSI/BFCRS and DSM-5 to describe signs of catatonia, a practical demonstration of how to elicit and document the DSM-5 diagnostic criteria for catatonia, and the 14-and 23-items in the BFCSI/BFCRS, respectively. Training also included how to uniformly capture the data on the data form and supervised practice sessions on each other and on practice participants under the guidance of the principal investigator.
An IRR with a Krippendorff α in the range of 0.61 and 0.8 during the practice scoring in at least one of the screening tools was considered acceptable for the assessors to proceed to the scoring of study participants for the broader prevalence study. 11 Assessments of the participants were performed on the same day for the IRR part of the study, or as close as possible. The definitions for catatonia that were inherent to the screening tools were accepted as the cut-off points for the diagnosis of catatonia, that is, the presence of two or more symptoms in the BFCSI/BFCRS and presence of three or more symptoms in the DSM-5. In cases where the assessors identified possible missed catatonia, the treating doctor was provided with any additional information found during the participant assessments to allow for a review of the patient's clinical case and management.

Setting
The setting was a 35-bed acute mental health unit in Dora Nginza Hospital, Nelson Mandela Bay Metro, which is a city with a population of 1.2 million people in the Eastern Province of South Africa. 12 The city has high unemployment and morbidity of mental illness rates. 13 Mental health services at the MHU include 24-hour care for persons who present with acute mental illness requiring inpatient treatment and electroconvulsive therapy for those who may need it.
The MHU receives referrals from other departments within the hospital as well as local clinics and district hospitals.

Sampling
Convenience sampling of all patients admitted to the MHU over the first six-month period of the study was undertaken.
The number of patients that were expected to be admitted during the first six months of the study period was approximately 130 based on unit admission figures over the previous six months. This was due to the restrictive effects of the COVID-19 outbreak on admissions rates and bed occupancy in the MHU. The restrictions were mainly because the new admissions 8 required screening for COVID-19, and the patients had to be admitted in separate spaces within the unit that would allow for social physical distancing. This resulted in a reduced bed capacity compared to the usual 35-bed occupancy and lower admission rates. Six months was chosen as the approximate duration it would take to recruit the 10 patients required for the IRR evaluation, given the effect of COVID-19 on admission rates.

Participants
Most people admitted to the Dora Nginza Hospital MHU were involuntary admissions under the Mental Health Care Act of 2002. 14 between the ages of 13 and 65 years and older due to the lack of child, adolescent, and geriatric inpatient-specific services in the region.

Data management and analysis
To determine the IRR, Krippendorff's α was computed for each screening tool.
Krippendorff's α allows multiple assessors to assess the same set of patients. 11 To make use of this method, each tool used to assess each patient for catatonia was recoded into a binary variable reflecting either the presence or absence of catatonia. A higher Krippendorff's α reflects a higher rate of agreement between the assessors. The presence of catatonia observed per tool was compared using Spearman's correlation coefficient (ρ). 15 Quantitative data on the assessment and presentation of catatonia in the 10 participants were summarized using descriptive statistics, with categorical variables presented using frequency and contingency tables. Descriptive statistics were calculated for the samples used to assess the IRR as well as on the rest of the samples for the prevalence study.

RESULTS
Eight (80%) of the 10 participants screened by the five assessors were 35 years or less in age, and 8 (80%) were male. At least six (60%) of the ten participants were found to have catatonia based on the BFCSI/BFCRS and DSM-5 criteria. The resulting Krippendorff's α values are listed in Table 1 Table 2. The highest correlation coefficients were observed between the BFCRS and BFCSI.

DISCUSSION
The assessment tools with the best IRR, as reflected by a Krippendorff's α of 0.798 and a correlation coefficient of 1, were the BFCSI and BFCRS. DSM-5 had the lowest Krippendorff's α of 0.565 and a correlation coefficient of 0.564 with both BFCRS and BFCSI.
Several studies have indicated the validity, ease of application, and good IRR of the BFCSI/BFCRS as an assessment tool for catatonia in the clinical setting. 1-5 7 8 None had evaluated the use of these screening tools in an African setting. Similarly, this study also reflects that BFCRS is the most sensitive instrument for picking up catatonia, even in a South African acute mental health unit setting. The DSM-5 on the other hand, had a much lower IRR score and correlated poorly with the other screening tools, indicating shortcomings when it comes to identifying catatonia. This implies that specifically for the assessment of catatonia, it may not be as useful in the clinical setting. [1][2][3][4] Thus, the findings of this study support the utility of the BFCSI/BFCRS as an assessment tool in an acute mental health setting and highlight the importance of using such tools over and above the DSM-5 when assessing patients with catatonia. The other important finding in our study is that it was possible to upskill mental health nurses to assess new admissions for catatonia after only one training session with a specialist psychiatrist, achieving a good level of IRR with a validated tool. This has potential implications for resource limited inpatient settings similar to the study site, which provide treatment for serious mental illness where catatonia might present. Since catatonia may also presents in Accident and Emergency, Internal medicine, and Neurology Departments, the implications of these findings may potentially be applicable in clinical environments beyond mental health units. This is especially important since catatonia may also present in delirious patients more likely to be seen in these clinical settings. 16

Study limitations
Limitations of this study include a small sample size, the number of assessment tools for catatonia, which did not include the Braunig's Catatonia Rating Scale or ICD-10, and the fact that the analysis only focused on the presence or absence of catatonia on admission and did not include cases where catatonia might have developed during the rest of the inpatient period.

CONCLUSION
The findings of this study indicate that the BFCRS and BFSCI are sensitive and reliable screening tools with good IRR that can be used by mental health professionals with only minimal training to screen for catatonia. The DSM-5 has notable deficits which limit its utility in the assessment of catatonia in the clinical setting. 4  Health granted ethics approval for the study and permission for data collection at the study site was obtained from the hospital manager. All patients admitted to the unit during the study period were provided with an information leaflet on the study in English or Xhosa and requested to take part. The capacity to consent was assessed using a simplified method based on the University of California Brief Assessment of Capacity to Consent. 17 18 This simplified method was a shortened version due to the COVID-19 outbreak, which required minimized contact time with participants or relatives to decrease the chances of infection.
For those who lacked the capacity to consent, proxy consent was sought from the closest relative. This ensured the protection of participant rights while still enabling the inclusion of participants or groups who could potentially benefit from scientific advances gained from research, as outlined in the Helsinki Declaration on ethical research. 19 The Department of Health Guidelines on Ethics in Health Research also supports this approach. 20 All data were anonymised and stored securely, with access to the research team.