The implementation of the nationwide out of hours phone number 1733. Analysis of eciency and safety.

Background In Belgium there is a problem with improper use of emergency services. The government installed an emergency number 1733 for out of hours care. Through a dry run test we learned that 10% of all calls were allocated to the protocol ‘unclear problem’. In only 10% of all cases, there was indeed an unclear problem. Methods This study aimed to determine whether the adjusted protocol ‘unwell for no clear reason’ led to a safer and more ecient referral and to evaluate the eciency and safety of the specic primary care protocols. The study was performed in a retrospective and prospective design and involved the community, patients, General Practitioner Cooperatives, emergency departments and telephone operators. Results During 6 months in 2018, 11622 calls to 1733 were registered. 756 of these calls were allocated to ‘unwell for no clear reason’. A random sample of 180 calls was re-listened. The eciency and safety of the protocol ‘unwell for no clear reason’ improved. The safety levels for under- and over-triage were not exceeded. The GP’s on duty judged that 9/10 of all patient encounters were correctly allocated. In less than 1% of all referrals, there was an under-triage. Conclusion This study demonstrated that the implementation of a 1733-operating system to triage patients to an adequate care level is successful if protocols, ow charts and care levels are well dened and operators well trained.


Introduction
In Belgium, there is an increasing problem with the improper use of emergency services (1,2). A study of the 'Federal Knowledge Centre for Healthcare' demonstrated that there is a need for a clear concept of an 'improper out of hours care use' and for the development of an effective triage system: a triage system to send the right patient at the right time to the right health care provider (3). There is a signi cant overuse of the emergency department encouraged by a free access without referral or nancial penalty. The inappropriate use of the emergency department occurs at the expense of the well-equipped and organised GPC (General Practitioner Cooperative) (4)(5)(6). In the follow up of the above mentioned report, the government installed in consent with the primary care and hospital setting a nationwide emergency number 1733 for unplanned non urgent out of hours care. Behind this number there is an operator directing the patient with an unplanned care need to the most appropriate level of care (6 levels with three intermediate levels): three levels of ambulance intervention, urgent or planned referral to out of hours primary care services or to planned care (https://www.health.belgium.be/en/health/need-call-doctor-call-1733). These operators are trained to follow the algorithms in a computer aided dispatching. These protocols are the result of a collaboration and consent between academic partners, the Federal Public Service Health, experts in general practice and in hospital emergency medicine.
To evaluate the use, e ciency and safety of the application of the protocols, a dry run test phase was implemented in a pre-determined and representative region of Flanders. During this test-phase, all patients were asked to call the 1733 emergency number in the presence of a researcher. The operator acted as in real life: he interrogated the patient following an instructive ow chart and referred according to the algorithm of the appropriate protocol to the appropriate level of care. From this study, we learned that in 10% of all calls the operator assigned the call to the protocol 'unclear problem'. In 10% of all these cases, there was indeed an unclear problem (eg. communication problem, multiple complaints, language problem). In the other cases there was no appropriate protocol available or there was a problem on operator level.
The writing and reviewing of triage protocols is a dynamic process with as main objectives to guarantee patients' safety and to control the expenses. The initial protocols (version 1.7.0) were adjusted and expanded with speci c primary care protocols (version 2.1.4). The operators were further trained to improve the use of the new protocols. The protocol 'unclear problem' was renamed as 'unwell for no clear reason'. The Belgian Counsel for Urgent Healthcare approved the nal version (4.0) of the protocol manual for general use in all 112 centres.
The aim of this research was: to determine whether the adjusted protocol 'unwell for no clear reason' led to a safer and more e cient referral of calls to the appropriate care level and to evaluate the e ciency and safety of (referral to) the speci c primary care protocols.

DESIGN
The study was performed in a retrospective and cross sectional design. The earlier protocol 'unclear problem' was adjusted and renamed to 'unwell for no clear reason'. The operator choses this protocol when the caller reports complaints or symptoms that do not t into any other protocol.
A protocol is e cient when it leads to the appropriate level of care: three levels of ambulance intervention, urgent or planned referral to out of hours primary care services or to planned care. A protocol is safe when the rate of under-triage is less than 15%. This maximum rate of acceptable under-triage was based upon a consensus between experts, the Intermediary Report Project 1733 Leuven-Tienen and relevant national international guidelines and research reports.
The caller is the person calling the national number 1733 and the patient is the help seeking party. The operator is the professional answering the 1733 calls. This operator follows an instructive owchart to question the caller instructed by the appropriate protocol and to refer the patient to the correct level of care. Each owchart starts with a set of medical terms de ned as keywords. The operator enters the keyword and picks the correct protocol. Every protocol refers to different levels of severity each leading to a level of care.
To determine whether the adjusted protocol 'unwell for no clear reason' leads to a safer and more e cient referral to the care level, randomly selected calls were re-listened by the researchers. The selected calls took place between May 4 and June 26 of 2018 and were labelled as 'unwell for no clear reason'. Each call was fully listened to by two researchers independently of each other. The researchers used a structured tool inventorying time of day, summary of patients' story, motivation of protocol choice and level of care, to assess the calls. The researchers picked the protocol of their choice and motivated this choice by a label: correct protocol, missing protocol, fail to recognize keyword, fail of the operator, communication problem, impossible to refer (too unclear call). The de nition of the labels was the result of expert consent and based upon the earlier pilot study.
To determine the e ciency of referral to the level of care, the data were analysed according to the 'Grounded Theory approach'. The researchers (n=5, General Practitioners in training and 2 graduated supervisors) discussed the completed assessments. In case of disagreement with the operators' choice of care level, the call was labelled as 'discordant'. In case the researcher did not reach consent on the correct care level, the call was present to the advisory group of the project 1733. If the operator referred to a lower care level, the call was labelled as 'under-triage'. If the operator referred to a higher care level, the call was labelled as over-triage. Since it concerned very few cases, discordancy was expressed by gures and no statistical analyses was performed.
In the second research question the e ciency and safety of the primary care protocols (PCP) was studied following the same procedure as prescribed above. For each of the 18 PCP's 15 calls were randomly selected within the period of 01-01-2018 until 30-06-2018. Second, all patients referred to these protocols were individually assessed by the GP on duty (Weekends from July 27-30, 2018 and February 8-11, 2019, 24 hour shift). The assessment took place when the patient presented at the GPC and before the consult started. Subsequently, the GP-assessor selected the appropriate level of care. This option was compared to the option the operator picked. The outcome of the assessment did not affect the provide care: all patients received the initially proposed care. E ciency of referral by the operator was measured by the level of concordance between the operator and the GP referral to a care level. Safety was measured by the occurrence of under-triage. Since it concerned very few cases, discordancy was expressed by gures and no statistical analyses was performed.

ETHICAL APPROVAL
This study was approved per research question by the Medical Ethical Committee of the University Hospitals of the KU Leuven (041C55FA505A80, 04324FEAA15C80, 047066BAC85E80). In case of patient inclusion or use of patient data, informed consent was asked.

Results
E ciency and safety of 'Unwell for no clear reason' (table 1) For the period of January 2018 to June 30 2018, 11622 calls were registered. 756 (6,6%) of these calls were allocated to the protocol 'unwell for no clear reason' of which a random sample of 180 calls was extracted and re-listened. 16 calls were excluded because of technical data base errors. In total 164 calls were included in the data-analysis. 60 (37%) calls took place in the morning, 45 (26,8%) in the afternoon, 44 (27%) in the evening and 15 (9,1%) during the night.
The label 'fail to recognize keyword' was allocated to 51 calls (31%), the label 'correct protocol' was allocated to 44 (27%) calls and the label 'missing protocol' to 37 (23%) calls . The label 'fail of the operator' was allocated to 23 (17%) calls of which 18 calls were labelled as 'communication problem'. No call was labelled as 'impossible to refer'.
When assessing the e ciency of allocation to a protocol, 104(64%) calls were labelled by the researchers as 'unwell for no clear reason'. Eight calls (5%) were allocated to the protocol 'dizziness', six calls to the protocol 'skin problems' and ve (3%) calls to 'airway problems'. All other calls (n=41) were allocated less than ve times to another protocol. 135 (82%) calls were referred to the out of hours primary care services and 23 (14%) calls were urgently referred to this care level. Planned care was advised in four (3%) calls. Two (1%) calls were referred to 'ambulance intervention'.
When assessing the safety of referral, the researchers referred 111 (68%) calls to the same level of care as the operator did and 53 calls were referred to another care level. 36 (22% of total) of these calls were recognized as 'over triage'. 17 (10% of total) calls were recognized as 'under triage.
The allocation to an inappropriate protocol led in 25 (15%) calls to a discordance of the referral to a care level between the operator and the researcher. In 12 calls the referral was labelled as 'over-triage' and the referral of 13 calls was recognized as 'under triage'. In 2 calls there was an 'under triage' of 2 levels.
The e ciency and safety of (referral to) the speci c primary care protocols (tables 2) For each of the 18 primary care protocols 15 calls were randomly selected for assessment. For seven protocols, less than 15 calls were available. In total, the researchers assessed 202 calls. 75 (37%) of the calls was recorded in the morning, 65 (32%) in the afternoon, 49 (24%) in the evening and 13 (7%) at night.
To determine the safety of referral to the PCP, the researchers assessed the allocation to the level of care.
175 calls (87%) were allocated to 'out of hours primary care services', 19 calls (9%) to 'urgent out of hours primary care services', 7 calls (3%) to 'ambulance' and 1 (1%) call to 'planned care'. The researchers referred 46 (23%) calls to another care level: in 22 (11%) calls there was under-triage and in 24 (12%) overtriage. The researchers disagreed about the appropriate level of care in 10 calls. These calls were re-listened and discussed until the team reached consensus: two calls remained undetermined due to a language issue and due to missing administration of referral. The researchers labelled 99 (50%) calls as 'correct protocol and correct care level' and 27 (13%) calls as inappropriate protocol and correct care level'. 57 (28%) calls were labelled as 'incorrect protocol' but correct care level. In 19 (9%) cases, the researchers labelled a call as 'inappropriate protocol and inappropriate care level'.
To determine the safety of referral to a care level using the PCP, 325 allocated patients received an assessment by the GP on duty (table 3). Seven assessments were incomplete and 68 assessments were completed after clinical judgement. In total 75 assessments were excluded for further analysis. 102 included assessments took place in the rst weekend and 148 in the second weekend.
After assessment by the GP on duty, 236 (94,4%) calls were correctly referred with a small non-signi cant difference between summer and winter. In two (0.8%) cases the patients needed referring to a higher level of care and in 12 (5%) cases a lower level of care.

Discussion
The e ciency and safety of the protocol 'unwell for no clear reason' improved after the adjustments to the primary care protocols. The number of 1733 calls referred to this protocol by the operator signi cantly decreased as compared to the rst trial and the concordance rate between allocation of a call to this protocol by operator and researcher was 2/3. This rate was higher than in the test phase and the rate of under-triage by the use of 'unwell for no clear reason' was 1/10. The concordance of allocation to the newly developed primary care protocols by operator and researcher was only 1/2 but the safety levels for under-and over-triage were not exceeded. The GP's on duty judged that over 9/10 of all patient encounters at the guard post were correctly allocated. In less than 1% of all referrals, there was an under-triage.
By improving the primary care protocols, there was a manifest decrease of the number of calls assessed as 'missing protocol'. In particular protocols on 'nose-ear-throat', 'skin problems' and 'musculoskeletal problems' were missing in the rst tested version. When assessing the allocation of calls to the protocol 'unwell for no clear reason' the researchers now missed the protocols 'fever adult', 'breast problems' and 'hypertension'. By developing and implementing these protocols, the number of allocation to 'unwell for no clear reason' will further decrease (7,8).
Strikingly, in one third of the referrals to 'unwell for no clear reason', the call was labelled as 'fail to recognize key word' by the operator. This rate was higher than in the test phase and could be explained by the lack of experience and con dence with the newly implemented protocols. Education and training of the operators is therefore important (9,10).
The implementation of primary care protocol seemed successful since 1733 allocated more than 9/10 of all calls to the GP guard post. Only 1% of all calls were referred to planned care. This low number is mainly due to a lack of experience with this care level, which did not exist before 1733 (11)(12)(13).
The rate of under-and over triage was acceptable. Over triage was more likely than under-triage and mainly due to referral to care level 'out of hours primary care services' instead of referral to 'planned care'.
Under triage may result in an unsafe situation (8,10,12). The number of under-triage was mainly caused by referral to 'out of hours primary care services' instead of 'urgent home visit'. In two cases, there was an unacceptable under-triage by referring the patient to 'out of hours primary care services' instead to 'ambulance'. The rate of under-triage by allocation to the protocol 'unwell for no clear reason' was below the limits set by the taskforce.
The main reason of discordance between operator and researcher in allocation to the primary care protocol was a fail of the operator and a fail to recognize the keyword. This was particularly the case when the age of the caller was important to allocate to a particular protocol (eg. child with fever). Operators often failed to question the context and the state of emergency to refer to the correct level of care. A structured and digitized algorithm could support the questioning by the operators and prevent from missing keywords or relevant information (7,8,14). As reported by other authors, operators seemed often confronted with dissatis ed callers who expected to receive a medical advice by telephone (10,15,16).
Public information campaigns are necessary to align the objectives of the 1733-system and the public expectations (13,17). Strikingly, the operators only once opted for the care level 'planned care' and also the researchers rarely opted this care level. This observation could be due to the apparent similarity between care level 'out of hours primary care services' and 'planned care'. Second, operators are in our health care tradition not con dent with this care level. Language barriers and missing of keywords in protocols were also a reason for incorrect referral. Probably, this is a point of attention that needs to be evaluated on regular base (10,18).

Strengths and limitations
The strength of this research lies in the fact that it started in a dry run phase and then after evaluation and adjustment, continued in a real life setting. Second, the researchers, practising GP's, took part in the development and the ne-tuning of the protocols.
An important limitation of the study is that the assessment of the operator intervention by re-listening the calls, was based upon the experience and competence of the researchers. Although a thorough reviewing and revision of the results was performed, errors or misinterpretations cannot be ruled out. Moreover, the researchers were trained GP's while the majority of the operators do not have a medical background.

Conclusion
This study demonstrated that the implementation of a central 1733-operating system to triage patients in an off duty period to an adequate care level is successful provided that protocols, ow charts and care levels are well de ned and operators well trained.