Usability testing of the Smart Work Injury Management (SWIM) system to assist case managers in predicting the work disability of injured workers

doi:10.21203/rs.3.rs-3909843/v1

Download PDF

Research Article

Usability testing of the Smart Work Injury Management (SWIM) system to assist case managers in predicting the work disability of injured workers

https://doi.org/10.21203/rs.3.rs-3909843/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Purpose Many countries have developed clinical decision-making support (CDMS) tools, such as the Smart Work Injury Management (SWIM) system in Hong Kong, to predict rehabilitation paths and address global issues related to work injury disability. This study aims to evaluate the accuracy of SWIM by comparing its predictions on real work injury cases to those made by human case managers, specifically with regard to the duration of sick leave and the percentage of permanent disability.

Methods The study analysed a total of 442 work injury cases covering the period from 2012 to 2020, dividing them into non-litigated and litigated cases. The Krustal-Wallis post hoc test with Bonferroni adjustment was used to evaluate the differences between the actual data, the SWIM predictions, and the estimations made by three case managers. The intra-class correlation coefficient (ICC) was used to assess the inter-rater reliability of the raters.

Results The Krustal-Wallis test revealed a statistically significant similarity between the predictions of SWIM and a case manager with about four years of case management experience on non-litigated cases. The predictions made by case managers with 9 and 20 years of experience were also significantly similar for all cases. However, SWIM’s prediction of the permanent disability percentage differed significantly from the predictions of the other groups.

Conclusions The study confirmed that SWIM could effectively simulate the predictions of a case manager with roughly four years of case management experience regarding sick leave duration. It is recommended that future research considers the influence of psycho-social factors on predictions as these are often factored in by experienced case managers.

Implications SWIM can aid case managers in estimating non-litigated cases, thereby addressing the shortage of human resources in Hong Kong and reducing inconsistencies in case estimations.

Clinical Decision-Making Support

Prediction

Artificial Intelligence

Worker’s Compensation

Work Disability Management

Prolonged worker absence from work due to injury is a global concern. According to the International Labour Organization [1], the total number of days lost in 64 surveyed countries increased from approximately 44 million in 2010 to approximately 62.5 million in 2020, representing a 42% increase over that period. From 2010 to 2016, the number of days lost in Hong Kong remained relatively stable at approximately 300,000 days per year [1]. In 2021, there were 30,185 non-fatal occupational injuries in Hong Kong [2]. This phenomenon of enormous loss of days has led to lost wages, lower productivity, and higher permanent disability. In addition, work injuries have created a public health issue and have indicated the need for injury prevention or health and safety reinforcement for workers in the future. The waiting list for medical consultations and rehabilitation treatments, especially with orthopaedic specialists, is always long in public hospitals. This affects the medical recovery of injured employees in terms of missing the “gold recovery period” for injuries. The accumulated demand for rehabilitation has placed an inevitable burden on public health systems. Therefore, policymakers, governments, and employers are keen to prioritize the rehabilitation management of work injuries and the societal impact of work disabilities in order to minimize the influence on both individuals and society as a whole.

To address this concern, various rehabilitation programmes have been implemented in recent decades. In most countries, governments have adopted the centralized case management approach, whereby all work injury cases are managed by government-funded companies. This approach can highly control and consolidate the rehabilitation service in a country and make better use of public funds in this area. In Hong Kong, there is no centralized model; cases are handled by individual insurance companies. Nevertheless, some steps have been implemented by the Hong Kong Government for work injury rehabilitation. For example, in March 2003, the Hong Kong Labour Department and the insurance industry implemented the Volunteer Rehabilitation Programme (VRP), which aims to provide injured workers with timely and free rehabilitation services for a speedier recovery and an early return to work under safe circumstances [3]. Other programmes include the Multi-disciplinary Orthopaedics Rehabilitation Empowerment (MORE) programme established in 2011 [4] and the recent Pilot Rehabilitation Programme for Employees Injured at Work [5] launched by the Labour Department in September 2022; these programmes adopt the case management and early intervention approach to better manage the medical recovery of injured Hong Kong employees by providing speedy private rehabilitation services.

Case Management in Hong Kong

Case management has a rich history dating back to 1863 in the United States in areas ranging from community health nursing to mental health, and it has progressively resulted in wide variations in models [6]. In 1995, Martin described case management as “a systematic approach to coordination of services to suitable clients through efforts of assessing providers, treatments, and developing treatment plans which improve quality and efficacy while controlling costs and monitoring outcomes” [7]. The case management model has been effective in reducing the number of lost days, medical costs, and claim costs [8] and most importantly has improved the continuity of case management and promoted the efficient and appropriate reintegration of employees into the workplace [7]. Therefore, over the last 20 years, many employers and insurers in Hong Kong have adopted the case management approach in managing work injury cases to reduce work disability in terms of sick leave days and permanent disability percentage (PD%). However, there are limitations of case management in Hong Kong. Hong Kong is a city with a high staff turnover rate, and this affects knowledge inheritance. Due to the high immigration rate in Hong Kong in the past few years and because most of the experienced case managers in Hong Kong have withdrawn from the work market for family reasons, it is common for most case managers in the market to be inexperienced and not to have the skills required to manage complicated cases. Due to this turnover and shortage of experienced case managers, there is a lack of formal training and insufficient learning time for new case managers. In addition, there is no regulation or standard regarding the qualification requirement for case managers in Hong Kong. Case managers are not required to have any recognized licence or academic qualification.

In receiving a work injury case report, a case manager is required to estimate a “reserve” on the basis of the available medical information. A “reserve” is defined as “funds which are set aside by an insurance company for the purpose of meeting obligations as they fall due. Such obligations would include liabilities for unearned premiums and the estimated costs of unpaid claims” [9]. A reserve will be reviewed and updated across the case journey on the basis of the development of the case, such as a deterioration in the progress of the case or the injured worker appointing a legal representative to proceed with a civil litigation claim. This reserving practice is a legislative requirement in Hong Kong to prevent insufficient funds being available to pay for the claims of the policyholders. The reserve estimation is directly related to the capability of the case manager to accurately estimate the number of sick leave days and the PD% of a case. Furthermore, insurance companies often have their own reserving guidelines and approaches. Therefore, the limitations of case managers in terms of their quality, capability, and lack of experience will affect the accuracy of reserve estimations. Moreover, the main objective of a case manager is to develop a rehabilitation treatment plan for the injured worker. An inaccurate estimation of the prospective outcome of the injury will have an adverse impact on the worker’s recovery path.

Clinical Decision-making Support Tools

The use of clinical decision-making support (CDMS) tools has become a leading response in tackling work disability and community health issues generally. CDMS systems have been developed in various medical areas, such as ambulatory/primary care [10] and genetics healthcare [11]. These CDMS tools have been found to have potential in producing statistically significant improvements in outcomes; nevertheless, further studies are needed to evaluate the results and to determine effective implementation. CDMS tools have been widely developed in many general medical areas, but not in work injury rehabilitation. In portfolio management, they have been one of the ways used to predict work disability and evaluate the results of rehabilitation programmes. Accurate and reliable innovative forecasting systems would add value to the whole process of case management, including the rehabilitation treatment plan, the case management strategy, and the reserve estimation to be budgeted. A few predictive models have been developed in the industry, but these models have not yet been widely used. One model is the Work Assessment Triage Tool (WATT), which is used in Canada to classify injured workers into appropriate rehabilitation programmes by machine learning [12]. In 2016, Qin et al. [13] conducted a validity study for WATT and suggested further research to provide evidence of its effectiveness. Another model is the gradient boosting machine (GBM), which is used in Korea to predict return to work (RTW) after sick leave [14]. In Hong Kong, several computer assisted therapy softwares have been developed and studied [15, 16]. Among them, the Smart Work Injury Management (SWIM) system is one of the CDMS tools that uses artificial intelligence to predict RTW trajectory and to provide advice on medical care and RTW interventions [17]. SWIM is a cloud platform containing a set of management tools for data storage, data analytics, and machine learning [18]. Unfortunately, there has been no validation study to show its accuracy. Therefore, the aim of this study is to evaluate the usability of SWIM as a clinical decision support tool to assist case managers in predicting the work disability of injured workers. Using real work injury cases, the study examines the accuracy of the predicted values of the SWIM system compared with the estimations of three case managers with different years of experience and backgrounds, which are presented as human estimations, on sick leave days and PD%. In addition, the study examines the differentiation and/or similarity between SWIM, case managers, and the real cases. The results will be used to assess SWIM’s readiness to be implemented in the industry and to identify its limitations.

Designs

This study used the severity model of the SWIM system (version 1.0) (Fig. 1) to predict sick leave days, PD%, Medical Assessment Board (MAB) days, whether a case requires high-level management (HLM), and return-to-work (RTW) days. The study only used the sick leave days and PD% results for analysis. SWIM requires five mandatory data points to produce a prediction: age, gender, injured body part, nature of injury, and cause of injury. Other optional information that can be inputted includes industry, position, and salary. Physical demand level, alertness, and frequency of manual handling operations are preset and appear on the mandatory data input. These preset fields can be adjusted manually on the basis of the particular situation of a case. In this study, we compared the sick leave days and PD% predictions of the SWIM system with the estimations of three case managers (CMs) with different levels of case management experience and backgrounds: CM-A, a female with 4 years of work injury case management experience and an academic background in marketing; CM-B, a female with 9 years of case management experience and a registered nurse; and CM-C, a female with approximately twenty years of experience and a legal background. CM-B and CM-C were working in the same company at the time of the data collection (Table 1). Since the total number of cases used was 442, only three case managers agreed to participate voluntarily in the study due to the long length of time required for the estimations.

Table 1

Demographic information of case managers.
	CM-A	CM-B	CM-C
Gender	Female	Female	Female
Age	35–40	35–40	45–50
Work injury case management experience (years)	4	9	20
Academic background	Marketing	Nursing	Legal
Company	Different from CM-B and CM-C	Same as CM-C	Same as CM-B

Participants

The data were extracted from the claims database of an insurance company in Hong Kong. A total of 546 claims covering the period between 2012 and 2020 were initially identified; 104 of these cases were not valid and were excluded because they were either fatal cases, cases with no claim form, or cases with insufficient information. The remaining 442 work injury cases (mean age = 45.9; male = 397 and female = 55) were adopted for this study. The data contained the following information about the cases: the demographic information (occupation, physical demand level, age at the date of accident, and gender) of the injured workers and information on alertness, injured body part, nature of the injury, cause of injury, frequency of manual handling operations, actual sick leave, and actual PD% (Table 2). PD% data were available for 267 of the 442 cases, and all cases contained data on the actual sick leave days taken by the injured workers. At the time of the data collection, these cases were at different stages. A total of 332 out of the 442 cases involved employees’ compensation litigation proceedings and 110 were non-litigation cases.

Table 2

Demographic and injury information of participants
	Validation data (n = 442)	Percent
Age at date of accident (years)	Mean (SD) = 45.9 (11.5)
Sex
Female	55	12.400
Male	387	87.600
Occupation
Air duct Fitter	1	0.002
Air-conditioning Engineer	6	0.014
Bar bender & fixer	26	0.059
Bricklayer	1	0.002
Carpenter	21	0.048
Cleaner	2	0.005
Concrete Worker	6	0.014
Construction Site Worker	142	0.321
Crane Operator	5	0.011
Curtain wall Technician	5	0.011
Driver	4	0.009
Electric Technician	21	0.048
Fitter	4	0.009
Foreman	19	0.043
Forklift Operator	1	0.002
Formwork Worker	10	0.023
Ganger	1	0.002
General Labourer	23	0.052
Glass panel Installer	1	0.002
Installation Worker	2	0.005
Landscape Attendant	1	0.002
Leveller	2	0.005
Machine Operator	3	0.007
Marble Worker	2	0.005
Mason	1	0.002
Mechanic	1	0.002
Metal worker	8	0.018
Nil	27	0.061
Painter	4	0.009
Plasterer	13	0.029
Plumber	13	0.029
Rigger	24	0.054
Scaffolder	18	0.041
Security Guard	1	0.002
Skilled Labourer	3	0.007
Steel Worker	6	0.014
Technician	4	0.009
Welder	10	0.023
Injured Body Part
Abdomen	1	0.002
Ankle	29	0.066
Back	72	0.163
Chest	19	0.043
Ear	1	0.002
Elbow	8	0.018
Eye	13	0.029
Face	1	0.002
Finger	67	0.152
Foot	26	0.059
Forearm	16	0.036
Hand/palm	22	0.050
Hip	2	0.005
Knee	17	0.038
Leg	28	0.063
Mouth/tooth	3	0.007
Neck	18	0.041
Pelvis/groin	2	0.005
Shoulder	20	0.045
Skull/scalp	9	0.020
Thigh	6	0.014
Upper arm	6	0.014
Nil	56	0.127
Nature of injury
Abrasion	3	0.007
Amputation	10	0.023
Burn	3	0.007
Concussion	1	0.002
Contusion & bruise	64	0.145
Crushing	11	0.025
Dislocation	5	0.011
Electric shock	2	0.005
Fracture	147	0.333
Freezing	1	0.002
Irritation	1	0.002
Laceration and cut	21	0.048
Multiple injuries	12	0.027
Others	15	0.034
Puncture wound	3	0.007
Sprain & strain	91	0.206
Nil	52	0.118

Procedures

All mandatory and available optional data were inputted into SWIM. The predicted sick leave days and PD% were recorded for the 442 cases as the experimental group. The same set of data was given to CM-A, CM-B, and CM-C, without knowing the results from SWIM, as the control group. The case managers estimated the sick leave days and the PD% on the basis of the case information provided. Then, the SWIM projected values were compared with the estimations made by the case managers and the actual cases.

The analysis was separated into a group of 110 non-litigated cases and a group of 332 litigated cases. There were 442 cases with data on sick leave days and 273 with PD% data. A litigated case was defined as a case where the injured worker had filed a civil litigation claim against their employer. First, the Shapiro-Wilk test was used to assess the normality of the dataset using Statistical Package for the Social Science (SPSS) version 28 [19]. The test indicated that the sick leave days and PD% data were not distributed normally. Therefore, the Krustal-Wallis post hoc test with Bonferroni correction was utilized to assess the significant difference between the actual cases, SWIM, and the case managers. The Kruskal-Wallis test is the non-parametric equivalent of a one-way ANOVA and is used for testing whether samples originate from the same distribution [20]. The test was followed by the post hoc Dunn-Bonferroni test to identify the pairwise significant difference from each group by adjusting the alpha level in reducing the type 1 error [21]. Dunn’s test has proved popular over the past two decades, and the test is frequently used with multiple comparison adjustments [20]. Thirdly, the intra-class correlation coefficient (ICC) was selected to test the inter-rater reliability of the predictions across different case managers. ICCs with 95% confidence intervals were calculated using IBM SPSS version 28.0.0.0 (190) based on a mean-rating (k = 3), absolute agreement, 2-way mixed-effects model (model 3) [21]. The ICC is a value between 0 and 1, where values below 0.5 indicate poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 indicate good reliability, and any value above 0.9 indicates excellent reliability [22, 23]. The findings of this study can be used to validate SWIM as a CDMS tool in Hong Kong and has a positive significant influence on work injury rehabilitation management by using artificial intelligence.

The Shapiro-Wilk test showed that the distributions were statistically significantly (p < 0.001) non-normal for all the data, including both sick leave days and PD%. Therefore, the Krustal-Wallis test was used to assess the difference between the actual data, the SWIM predictions, and the estimations produced by the three case managers. The null hypothesis for the Krustal-Wallis test is that the population medians of the groups are equal. A rejection of the null hypothesis signifies that two or multiple groups are significantly different. Post hoc test with Bonferroni adjustment was adopted for making pair-wise comparisons [20].

Sick Leave Days

The test was applied to two separate groups: 1) non-litigated cases and 2) litigated cases. For the non-litigated cases, the Krustal-Wallis test showed that three pair-wise comparisons failed to reject the null hypothesis and indicated that the populations of these pairs were statistically significantly the same. The first group was SWIM and CM-A (χ² (4, N = 110) = 14.145, p = 0.507, Bonferroni adjustment (b.j.) = 1.000, the second group was the actual data and CM-B (χ² (4, N = 110) = 34.145, p = 0.109, b.j. = 1.000), and the last group was CM-B and CM-C (χ² (4, N = 110) = 26.464, p = 0.215, b.j = 1.000). For the litigated cases, only CM-B and CM-C failed to reject the null hypothesis (χ² (4, N = 332) = 53.104, p = 0.153, b.j = 1.000).

Permanent Disability Percentage

The SWIM system’s predictions on PD% demonstrated statistically significant differences across all groups, irrespective of whether the cases were litigated or non-litigated. However, the actual data from non-litigated cases did not reject the null hypothesis when compared with CM-A (χ2 (4, N = 273) = -14.467, p = 0.289, b.j = 1.000) and CM-B (χ2 (4, N = 273) = -19.261, p = 0.158, b.j = 1.000). The statistical similarity in PD% among CM-A, CM-B, and CM-C was significant in their respective comparisons (CM-A & CM-B: χ2 (4, N = 273) = -14.467, p = 0.289, b.j = 1.000; CM-A & CM-C: χ2 (4, N = 273) = -26.522, p = 0.052, b.j = 0.521; CM-B & CM-C: χ2 (4, N = 273) = -21.728, p = 0.112, b.j = 1.000). With regard to the litigated cases, both CM-B and CM-C (χ2 (4, N = 273) = -29.888, p = 0.324, b.j = 1.000), and the actual data and CM-C (χ2 (4, N = 273) = 47.211, p = 0.119, b.j = 1.000) exhibited identical population means.

Using the ICC, the inter-rater reliability of the SWIM predictions was tested across different case managers who were assessing the same set of cases. The test showed moderate reliability between SWIM and CM-A (ICC = 0.637, p < .001), CM-A and CM-C (ICC = 0.642, p < .001), and CM-A and CM-B (ICC = 0.581, p < .001) for non-litigated cases. For non-litigated cases, CM-B and CM-C demonstrated good reliability (ICC = 0.837, p < .001), which was also the case for litigated cases (ICC = 0.845, p < .001).

Injuries at the workplace constitute a common and substantial public health problem with major economic consequences worldwide [24]. As time off work due to disabling injuries increases, injury-related costs such as indemnity payments, medical and legal expenses, and employee substitution costs rise. In the case journey of a work injury case, a disability prognosis is essential as this will impact the expected medical intervention from medical professionals, the resource allocation from an operational perspective, or the compensation reserved by insurance companies [25, 26]. The existing prognosis forecasting in industry relies on human decisions based on different companies’ internal guidelines and the experience of doctors, rehabilitation professionals, or case managers. However, there are limitations of case management quality in Hong Kong, such as the high turnover rate of staff, the lack of experience, the lack of formal training, and the lack of a regulation or standard regarding the qualification requirement for case managers. This creates inconsistency, with potential over- or under-estimation of the risk, which will affect the relevant organizations from the long-term portfolio management perspective. Importantly, inaccurate estimation of injury recovery will have adverse impact on the case management and the treatment plan developed for injured workers [27]. Otherwise, this will affect the expectation on recovery outcome and can lead to misallocation of medical resources and postponement of treatment. Clinical decision-making tools using artificial intelligence to build a disability forecasting system can solve this problem by increasing consistency and controlling the risk of mis-assessment on medical prognosis effectively [28, 29]. In the journey of a work injury case, the injured worker will always be influenced by many psycho-social factors: for example, the injured employee may adopt a sick role. The Parsonian Model has outlined the role of a sick person be exempted from responsibility for the incapacity and from normal social role obligations. This incapacity as illness provides a legitimate basis for the sick individual’s exemption from normal role and task obligations [30]. This causes the worker to be on sick leave for longer. Eggert found that frustration, depression, discrimination, and obstacles in understanding how the workers’ compensation system works affect workers’ ability to return to work [31]. Booth-Kewley’s 2013 study found that recovery expectations was a key predictor of injury recovery and suggested early interventions to modify workers’ expectations [32]. In 1995, Rohling conducted a meta-analytic review of financial compensation (sometimes called compensation neurosis) and the treatment of chronic pain and found that compensation is related to increased reports of pain and decreased treatment efficacy [33]. Compensation neurosis is a controversial concept that the psychological and physical symptoms that may arise in injured workers who are involved in litigation for the personal injury, and that may be influenced by their expectations of compensation [34, 35].

Sick Leave Days

The results of this study showed that for non-litigated cases, SWIM simulated predictions of sick leave days in line with the estimations of CM-A, who had approximately four years of case management experience. However, this was not the case in regard to litigated cases. As a case progresses, numerous psycho-social factors affect its progress, and experienced case managers consider these potential factors when making estimations based on injury severity and demographic information [36, 37]. The reserving guidelines of their companies also influence their estimations. A further analysis between non-litigated and litigated cases, which this case legal or non-legal status was unknown to the case managers during the experiment, showed that the mean sick leave days estimated by CM-A were 109 days for non-litigated cases and 97 days for litigated cases, compared to SWIM’s predictions of 110 and 120 days, respectively. There was a noticeable difference between the estimations of CM-B and CM-C: CM-B estimated an average of 190 days for non-litigated cases and 225 days for litigated cases, while CM-C estimated an average of 169 days for non-litigated cases and 213 days for litigated cases. The results suggest that CM-A may not have considered any psycho-social factors when making estimations as the mean was similar between non-legal and legal cases in her estimations. On the other hand, the results showed that CM-B’s and CM-C’s estimations were significantly similar regardless of case status. This could be because case managers B and C were working in the same company and using the same reserving guidelines, and thus their case management approaches should be similar.

The results suggested that compared to CM-A, SWIM predicted a higher number of sick leave days for litigated cases but still could not simulate CM-B and CM-C, who had more experience. The results of the Krustal-Wallis test were consistent with the ICC test on inter-rater reliability, showing that SWIM and CM-A were moderately reliable (ICC = 0.637, p < .001) for non-litigated cases, while CM-B and CM-C were strongly reliable across all cases (ICC = 0.837, p < .001 non-litigated; ICC = 0.845, p < .001 litigated).

Interestingly, it was found that the estimation of CM-B (mean = 190) was significantly similar to the actual data (mean = 271) for non-litigated cases. This finding revealed that the mean of CM-B’s sick leave estimation was higher than that of CM-C for both litigated and non-litigated cases, even though their results were significantly similar. This suggests inconsistency among case managers, even when they adopt similar estimation approaches, and that their differing levels of experience can still yield varied results. Therefore, a clinical decision-making tool can address this inconsistency, which encourages further research into developing an accurate and validated system for work injury rehabilitation prediction similar to that of an experienced case manager.

Permanent Disability Percentage

It was unfortunate to find that SWIM’s predictions on PD% were statistically significantly different (p = < 0.001) across the actual data and from any of the case managers’ predictions under the Krustal-Wallis post hoc test. The ICC result was consistent with the findings of the Krustal-Wallis test, showing that the average measures were close to zero. The ICC also illustrated that CM-A was moderately reliable with CM-B (ICC = 0.682 non-legal; ICC = 0.559 legal, p = < 0.001) and with CM-C (ICC = 0.663 non-legal; ICC = 0.602 legal, p = < 0.001). CM-B and CM-C had good reliability for all cases (ICC = 0.715 non-legal; ICC = 0.795 legal, p = < 0.001). Therefore, the results suggested that further development or validation is required for the PD% predictions of SWIM.

Upon further analysis of the estimation pattern by nature of injury, the deviation of SWIM estimations from one injury type to another did not vary as much as those of the case managers, particularly CM-B and CM-C (Fig. 2). The boxplot in Fig. 2 is based on the Actual SL days. The most prevalent injury within the dataset of this study was fractures, accounting for 147 out of the 422 cases. The findings revealed a clear pattern indicating that the predictions made by SWIM and CM-A were similar but not accurate compared to CM-B and CM-C when compared to the actual outcomes of the cases. As the cases progressed, various psycho-social factors emerged that could impact the progress of the cases and potentially lead to an extension of sick leave. These factors included cases becoming legal matters, the injured employees maintaining a belief that they were unfit for work, or developing a compensation-oriented mindset rather than a recovery-oriented one. It is important to note that the case managers in this study conducted the initial estimation of sick leave without considering any psycho-social factors. Hence, it is expected that the actual data deviates from the predictions made by SWIM and other human case managers. The accurate estimation of prognosis is highly dependent on experienced judgment and the identification of high-risk factors that may deviate the course of individual cases while more case information is obtained from the injured employee or by the case development facts. This limitation in predicting the actual sick leave duration suggests the need for further research into the impact of psycho-social factors on rehabilitation prognosis.

The other observation in Fig. 2 is CM-B and CM-C estimated a higher average number of sick leave days for non-musculoskeletal injuries such as burns, concussions, or electric shocks, and severe injuries such as amputations, dislocations, and multiple injuries. The SWIM result was only slightly higher than the CM-A estimation, which aligns with this study’s finding that the estimations of SWIM and CM-A were statistically significantly similar. The pattern displayed in Fig. 2 also suggests that SWIM may have limitations in reflecting the actual risk for non-musculoskeletal and severe injuries. This is likely because insurance companies typically appoint rehabilitation service providers for musculoskeletal injuries rather than for other injuries, and therefore, the number of cases of the validation dataset had limited number on the other type of injuries. Alternatively, the dataset of SWIM may lack sufficient data to generate an effective algorithm for non-musculoskeletal cases.

Figure 3 shows that the average PD% predicted by SWIM did not vary much across different injury types, which is consistent with the Krustal-Wallis and ICC results showing that it was significantly different from other groups. The boxplot in Fig. 3 is based on the Actual PD%. Different cases should vary in PD% on the basis of demographic information and injury severity; however, SWIM’s predictions did not demonstrate this.

In conclusion, SWIM can simulate the prediction capabilities of a case manager with approximately four years of case management experience regarding sick leave days for non-litigated cases. However, it requires further development to simulate the prediction capabilities of more experienced case managers, particularly in relation to sick leave certification or PD% for all types of cases. It is crucial to consider psycho-social factors as they may be one of the factors that more experienced case managers consider when estimating injury prognosis. Case management is a complex process that involves numerous human factors and varying reactions from injured workers. The rehabilitation journey for each work injury case is unique; thus, continued efforts to simulate human estimation in this area are strongly recommended.

Limitations

The development of SWIM provides a valuable tool for clinical decision-making in Hong Kong, particularly for predicting work injury rehabilitation and sick leave days in non-litigated cases. However, there is room for SWIM to improve its accuracy in replicating the expertise of more experienced case managers, thereby addressing the resource shortage in the market. Despite the absence of psycho-social factors in the data provided to case managers, these factors are often considered by case managers due to their historical case handling experience. Given the jurisdiction of Hong Kong, it is common for injured workers to file litigation claims, which typically have a significant impact on return to work and compensation. Therefore, it is recommended that clinical decision-making tools in Hong Kong incorporate psycho-social factors, including litigation risk, when devising treatment plans. Medical treatment outcomes are notoriously poor in patients with pending litigation following disability claims, especially those covered by workers’ compensation programmes [38]. Consequently, this study suggests further research on work injury rehabilitation predictions, specifically examining or considering psycho-social factors to address high-risk factors.

Work injury rehabilitation is a global concern, and the objective of this study was to validate the utility of the SWIM system, a clinical decision-making tool, in Hong Kong. This tool aids case managers in predicting the work disability of injured workers. The study has confirmed that SWIM can simulate a case manager with approximately four years of case management experience, thereby assisting junior case managers in estimating the recovery duration for a particular injury and for an injured worker with a particular demographic background. This result addresses the significant practical issues of the shortage of case managers and the high turnover rate in Hong Kong. However, SWIM has limitations in terms of matching the prediction skills of experienced case managers and estimating the percentage of permanent disability. This may necessitate further development of the system’s algorithm or additional validation with other data to investigate the reasons for invalidation. Psycho-social factors are often considered by experienced case managers, and thus these factors could be a direction for future research on clinical decision-making tools.

Statements & Declarations

Funding

The author declares that no funds, grants, or other support were received during the preparation of this manuscript.

Competing interests

The author has no relevant financial or nonfinancial interests to disclose.

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Ethics approval and informed consent

This study adopted data without identifiable information and did not have any interaction with human subjects as it was not considered human subject research. The Ethics Committee for Human Subjects at the Hong Kong Polytechnic University has confirmed that no ethical approval is required, and no informed consent was needed.

ILOSTAT data tools to. find and download labour statistics. In: ILOSTAT. https://ilostat.ilo.org/data/. Accessed 2 Dec 2023.
Legislative Council. Hong Kong’s occupational safety performance in the first half of 2022. https://www.legco.gov.hk/yr2023/english/panels/mp/papers/mp20230221cb2-138-5-e.pdf. 2023. Accessed 2 Dec 2023.
Labour Department of Hong Kong. Voluntary rehabilitation programme for employees injured at work. https://www.labour.gov.hk/eng/public/ecd/WorkInjuries2.htm. Accessed 2 Dec 2023.
Law SW, Szeto GP, Chau WW, Chan C, Kwok AW, Lai HS, Lee RK, Griffith JF, Hung LK, Cheng JC. Multi-disciplinary Orthopaedics Rehabilitation Empowerment (MORE) program: a new standard of care for injured workers in Hong Kong. J Back Musculoskelet Rehabilit. 2016;29(3):503–13.
Pilot rehabilitation programme for employees injured at work. https://www.prp-wiro.gov.hk/tc/index.htm. Accessed 2 Dec 2023.
Kersbergen AL. Case management: a rich history of coordinating care to control costs. Nurs Outlook. 1996;44(4):169–72.
Dees JP, Anderson N. Case management. AAOHN J. 1996;44(8):385–90.
Iles R, Wyatt M, Pransky G. Multi-faceted case management: reducing compensation costs of musculoskeletal work injuries in Australia. J Occup Rehabil. 2012. https://doi.org/10.1007/s10926-012-9364-2.
Dooley Lucenti. Insurance reserves: what are they and how do they work? https://dooleylucenti.ca/insurance-reserves-what-are-they-and-how-do-they-work/.
Bryan CJ, Boren SA. The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inf Prim Care. 2008;16(2):79–91.
Sebastian A, Carroll JC, Oldfield L, Mighton C, Shickh S, Uleryk E, Bombard Y. Effect of genetics clinical decision support tools on health-care providers’ decision making: a mixed-methods systematic review. Genet Med. 2021;23(4):593–602.
Gross DP, Steenstra IA, Shaw W, Yousefi P, Bellinger C, Zaïane O. Validity of the work assessment triage tool for selecting rehabilitation interventions for workers’ compensation claimants with musculoskeletal conditions. J Occup Rehabil. 2019;30(3):318–30.
Qin Z, Armijo-Olivo S, Woodhouse LJ, Gross DP. An investigation of the validity of the Work Assessment Triage Tool clinical decision support tool for selecting optimal rehabilitation interventions for workers with musculoskeletal injuries. Clin Rehabil. 2015;30(3):277–87.
Na K-S, Kim E. A machine learning-based predictive model of return to work after sick leave. J Occup Environ Med. 2019;61(5):191–99.
Ng HF, Cheng AS. Reminiscence therapy in virtual reality: A virtual reality application with local content to help mild dementia. InAsian J Scholarsh Teach Learn 2023 (pp. 1–19).
Bu X, Ng PH, Tong Y, Chen PQ, Fan R, Tang Q, Cheng Q, Li S, Cheng AS, Liu X. A mobile-based virtual reality speech rehabilitation app for patients with aphasia after stroke: development and pilot usability study. JMIR Serious Games. 2022;10(2):e30196.
Cheng AS, Ng PH, Sin ZP, Lai SH, Law SW. Smart Work Injury Management (SWIM) system: artificial intelligence in work disability management. J Occup Rehabil. 2020;30(3):354–61.
Ng PH, Chen PQ, Sin ZP, Lai SH, Cheng AS. Smart Work Injury Management (SWIM) system: a machine learning approach for the prediction of sick leave and rehabilitation plan. Bioengineering. 2023;10(2):172.
King AP, Eckersley RJ. Statistics for biomedical engineers and scientists: how to visualize and analyze data. 1st ed. Elsevier: Academic Press; 2019.
Ostertagová E, Ostertag O, Kováč J. Methodology and application of the Kruskal-Wallis test. Appl Mech Mater. 2014;611:115–20.
EasyMedStat. Dunn-Bonferroni post-hoc test. n.d. https://help.easymedstat.com/support/solutions/articles/77000536997-dunn-bonferroni-post-hoc-test#:~:text=The%20purpose%20of%20the%20Dunn,pairs%20of%20means%20being%20compared.
Bobak C, Barr P, O’Malley A. Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales. BMC Med Res Methodol. 2018;18:93.
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–63.
Tompa E, Mofidi A, van den Heuvel S, van Bree T, Michaelsen F, Jung Y, Porsch L, van Emmerik M. Economic burden of work injuries and diseases: a framework and application in five European Union countries. BMC Public Health. 2021;21(1):1–0.
Snoeck-Krygsman SP, Schaafsma FG, Donker-Cools BH, Hulshof CT, Jansen LP, Kox RJ, Hoving JL. The perceived importance of prognostic aspects considered by physicians during work disability evaluation: a survey. BMC Med Inf Decis Mak. 2022;22(1):25.
Cancelliere C, Donovan J, Stochkendahl MJ, Biscardi M, Ammendolia C, Myburgh C, Cassidy JD. Factors affecting return to work after injury or illness: best evidence synthesis of systematic reviews. Chiropr Man Ther. 2016;24:1–23.
Trippolini MA, Young AE, Pransky G, Elbers NA, Lockwood K, Cameron ID. Beyond symptom resolution: insurance case manager’s perspective on predicting recovery after motor vehicle crash. Disabil Rehabil. 2021;43(4):498–506.
Harish V, Morgado F, Stern AD, Das S. Artificial intelligence and clinical decision making: the new nature of medical uncertainty. Acad Med. 2021;96(1):31–6.
American Hospital Association (AHA). How AI Is Improving Diagnostics, Decision-Making and Care. AHA [Internet]. [cited 2024 Jan 16]. Available from: .
Segall A. The Sick Role Concept: Understanding Illness Behavior. J Health Soc Behav. 1976;17(2):162–9.
Eggert S. Psychosocial factors affecting employees’ abilities to return to work. AAOHN J. 2010;58(2):51–5.
Booth-Kewley S, Schmied EA, Highfill-McRoy RM, Sander TC, Blivin SJ, Garland CF. A prospective study of factors affecting recovery from musculoskeletal injuries. J Occup Rehabil. 2014;24(2):287–96.
Rohling ML, Binder LM, Langhinrichsen-Rohling J. Money matters: a meta-analytic review of the association between financial compensation and the experience and treatment of chronic pain. Health Psychol. 1995;14(6):537–47.
Hall R, Hall R. Compensation neurosis: A too quickly forgotten concept? J Am Acad Psychiatry Law. 2006;34(3):313–25.
Mendelson G. Compensation Neurosis’ Revisited: Outcome Studies of the Effects of Litigation. Jornal of Psycosomatic Research. 1995;39(6):695–706.
ECCM. Understanding the Mental Health Case Management Process. ECCM [Internet]. [cited 2024 Jan 16]. Available from: .
Nelson K, Albrets P. Strengthening Case Management Practice: The Application of Psychosocial Acuity Scales. ACMA [Internet]. 2020 Mar 19 [cited 2024 Jan 16]. Available from: .
Bellamy R. Compensation neurosis: financial reward for illness as nocebo. Clin Orthop Relat Res. 1997;336:94–106.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
01 Mar, 2024
Reviews received at journal
29 Feb, 2024
Reviewers agreed at journal
05 Feb, 2024
Reviewers invited by journal
03 Feb, 2024
Editor assigned by journal
30 Jan, 2024
Submission checks completed at journal
30 Jan, 2024
First submitted to journal
30 Jan, 2024

You are reading this latest preprint version

Usability testing of the Smart Work Injury Management (SWIM) system to assist case managers in predicting the work disability of injured workers

Status:

Version 1

Abstract

Figures

Introduction

Method

Designs

Participants

Procedures

Results

Discussion

Limitations

Conclusion

Declarations

References

Additional Declarations

Status:

Version 1