All figures presented in this section are available as interactive and adjustable graphs within the dashboard platform and can be trialed with the web application which uses synthetic data (Supplementary material).
A summary of acute neurosurgical referrals in the post-lockdown period
10,033 acute referrals were made to our neurosurgical center between March, 2020 to October, 2021 (female = 4938, mean age [SD] = 61.1 years [18.8]). As would be expected, age and gender distribution varied widely according to diagnosis (Figure 2). For example, patients with a subdural hemorrhage presented with a mean age of 76.8 years [13.6] and male bias (male = 68.3%) as compared to patients classified as being suspected of cauda equina syndrome (mean age = 53.5 years [17.9], male = 43.4%).
The majority of referrals were classified as a brain tumor, degenerative spine or neurovascular diagnosis (Figure 3A) in line with the center’s main subspecialties. 96% of referrals were stated as an ‘emergency’ or ‘urgent’ by the referring team (Figure 3B) and 79% of referrals were made by a junior registrar or intern (Figure 3C). In terms of how referrals were triaged, 9.5% of referrals were accepted for immediate hospital transfer, 1% were placed on a transfer wait-list and 6.3% were assigned to outpatient review. 36.4% of referrals required additional clinical or imaging information from the referrer in order to make a triage decision. 32.1% of referrals were completed by triaging to conservative treatment or by offering only advice (Figure 4).
Weekly referral timing was found to be concentrated between 2-6PM on weekdays (Figure 5A), particularly for brain tumor referrals. Referrals for other high-volume categories such as degenerative spine and neurovascular diagnoses were more distributed but still significantly less over the weekend (Figure 5B, Supplementary Table 1).
During the aforementioned time period, referrals were received from 116 hospital sites and clinical institutions from across the U.K. (Figure 6A). Five hospitals in the Greater London catchment area accounted for more than 70% of overall referral volume (Figure 6B).
Choice of forecasting algorithm
Prioritizing computational time and test performance, Prophet was selected as the dashboard forecasting algorithm of choice (Figure 7). Although the CNN-LSTM algorithm demonstrated better performance across cross-validation scoring metrics, it was found to require a longer computational training and fitting time making it unsuitable under real-world computational constraints. ARIMA models are often considered as a benchmark model in forecasting 21. Here, with the addition of STL and an auto-hyperparameter tuning function, the cross-validation performance was comparable with Prophet, however its test performance was worse across time periods and was also marginally slower.
Change in referral volumes
Weekly referral volumes were compared between the first 6 months after the announcement of the U.K. Covid-19 pandemic lockdown 22 as compared to the same time frame after one year. There was a significant increase between these periods, mainly driven by an increase in spinal referrals (Table 1) which include spinal trauma, suspected cauda equina syndrome and degenerative diagnoses. Out-of-sample forecasting by all three time-series algorithms using all available training data demonstrated a consistent increasing long-term referral trend (Figure 7B, Supplementary Figure 1)
Table 1. Median weekly volumes and group-wise differences between the first 6 months of lockdown (March to August, 2020) and the same corresponding months after one year.
All p-values are Bonferroni multiple comparison corrected following univariate Mann-Whitney U tests.
(NS = not significant; *Aggregate of cranial trauma, subdural hemorrhage, stroke, brain tumor, neurovascular disease, congenital diseases and hydrocephalus diagnoses; ** = spinal trauma, cauda equina syndrome, degenerative spine and spinal tumor diagnoses; † = central nervous system infection, non-neurosurgical diagnosis)
Diagnosis
|
Early Covid-19 period median weekly volume
|
Late Covid-19 period median weekly volume
|
Difference
|
p
|
All
|
104
|
121
|
17
|
0.02
|
Cranial*
|
59
|
62
|
3
|
NS
|
Spinal**
|
34
|
44
|
10
|
0.02
|
Other†
|
8
|
13
|
5
|
<0.0001
|
Usability, feasibility and acceptability
20 participants were recruited for feasibility testing, including 5 neurosurgical consultants, 12 registrars and 3 members of management or administration staff. All were blinded to the development of the dashboard. Table 2 lists the average SUS, AIM and FIM scores among participants. An SUS score of 70 or above has previously been defined as a threshold for good usability 16. In this study, all user groups had mean SUS scores above this benchmark and high mean acceptability (AIM) and feasibility (FIM) scores were also recorded.
Analysis of user feedback explored possible reasons why the dashboard scored well (Supplementary Results, Supplementary Table 2). In brief, users highlighted the figures and interactivity as particularly useful features and felt that the dashboard would be useful to explore referral data, identify current areas for service improvement and suggest future directions for research. The use of time-series forecasting was commented as useful in anticipating service demand. In contrast, users expressed concerns regarding how the dashboard would be hosted and wished for additional functionality to review the data in more detail.
Table 2. Usability, feasibility and acceptability scores among main user groups
(SUS = System Usability Scale, scored out of 100; AIM = acceptability intervention measure, scored out of 5, FIM = feasibility intervention measure, scored out of 5; SE = standard error)
User group
|
n
|
SUS mean [SE]
|
AIM mean [SE]
|
FIM mean [SE]
|
All
|
20
|
77.1 [3.0]
|
4.7 [0.2]
|
4.6 [0.2]
|
Registrar / Resident
|
12
|
78.0 [3.1]
|
4.8 [0.1]
|
4.7 [0.1]
|
Consultant / Attending
|
5
|
74.2 [6.8]
|
4.4 [0.4]
|
4.2 [0.4]
|
Management & Administration
|
3
|
78.3 [10.4]
|
4.9 [0.1]
|
5.0 [0]
|