Setting, and design
We will perform a prospective diagnostic cohort study in the Netherlands in four GP cooperatives (Ede, Den Bosch, Uden and Oss) for out-of-hours primary care. The cooperatives serve a total of approximately 830,000 inhabitants in a mixed urban, suburban and rural area. The cooperatives are based in or adjacent to regional hospitals.
Patients
Patients will be recruited during out-of-hours home visits by GPs. Patients only receive home visits when they have acute medical complaints that cannot wait until the next working day and they are not able to visit the GP cooperative location for a clinic consultation. This is decided after telephone assessment by a triage nurse based on the Netherlands Triage System (NTS).16 All acutely ill adult patients ≥18 years with fever, confusion or general deterioration or otherwise suspected of a serious infection are eligible for inclusion. Exclusion criteria are: 1) Non-infectious cause of the acute complaints (e.g. stroke or myocardial infarction); 2) Hospitalisation within seven days before the home visit; 3) Condition that requires secondary care assessment in case of any signs of systemic infection (e.g. chemotherapy with possible neutropenia); 4) Terminal illness or other reason not to refer the patient to a hospital despite presence of a life-threatening condition.
Candidate predictors
We selected nine clinical features and three blood tests as candidate predictors for the development of the clinical prediction model (Table 1). Parameters of widely used scoring systems such as the SIRS, qSOFA and National Early Warning Score (NEWS)17 were considered, as well as clinical features used in guidelines such as the Netherlands Triage Standard (NTS) and NICE Sepsis guideline.18 Candidate predictors were considered when they were expected to have diagnostic value for sepsis, based on previous studies, and can be easily and objectively measured by GPs.
Candidate blood tests had to be currently used in the hospital setting for the diagnosis and/or prognosis of sepsis, and, preferably, to be available as a point-of-care test for reasons of implementation. CRP and lactate measurement are part of the standard care in patients with suspected sepsis during assessment in the Emergency Department (ED) in the Netherlands. Procalcitonin (PCT) is not routinely measured in most hospitals, but we decided to include PCT as a candidate predictor as PCT might be superior to CRP,19 and the NICE sepsis guideline specifically recommends to further evaluate PCT in sepsis research.
Outcome measures
The primary outcome measure is sepsis within 72 hours of inclusion. This will be determined by an expert panel using the Sepsis-3 criteria.1 In this consensus definition, sepsis is defined by an increase of two SOFA-points (Table 2) due to infection.
To limit the workload for the experts, we will install three expert panels, each comprising a GP, an emergency physician, and an internist(-intensivist). Each case will be assessed by one panel. All relevant information from medical records from the GP and the hospital when applicable will be presented to the panel. If there is no consensus on the primary outcome, the case will be discussed in a face-to-face consensus meeting with all three experts to determine the final outcome. Interobserver agreement between the three panels will be assessed in a selection of 10% of the cases that will be assessed by all panels. Besides the dichotomous primary outcome “sepsis within 72 hours”, the likelihood of sepsis will be assigned a numerical score between 0 and 10. This gives information on remaining uncertainty regarding sepsis classification, providing insight in the degree of bias that may be introduced when calculating diagnostic accuracy measures using dichotomous sepsis classification.20 Furthermore, the need for hospital treatment is scored between 0 and 10 by the expert panel as a secondary outcome. An average score above 5 will be regarded as a patient that should best be referred to the hospital immediately by the GP and a score ≤ 5 as a patient that does not have to be referred immediately.
Other outcome measures are hospitalisation (length of stay and type of care: ICU or regular ward), maximum SOFA score in the first 72 hours, 30 day all-cause mortality, final diagnosis, and medical costs.
Study procedures
Study period
The inclusion period is from June 2018 until April 2020. If in April 2020 the required sample size is not reached, the patient inclusion will be prolonged until the minimum required number of events has been reached. End of follow-up is 30 days after inclusion of the last patient.
Procedure during home visit
All patients receive usual care. Patients will be screened for eligibility during home visits by the attending GP of the GP cooperative. Verbal informed consent is obtained from the patient or his legal representative. The GP is (routinely) accompanied by a chauffeur during the home visit. The chauffeurs are used to practically assist the GP during the visit. Portable monitors (Philips intelliVue MP2 or X2) will be available to record peripheral oxygen saturation, automated blood pressure and heart- and respiratory rate by three lead electrodes on the chest.
The GP records the assessment of the candidate predictors in a case report form. In addition, the GP records if he/she has a gut feeling that “something is wrong” and provides the likelihood of presence of sepsis at inclusion on a scale from 0 to 10.
All study materials will be taken to the patient’s home in a study bag. The venous samples will be collected by either the GP or an on-call laboratory assistant or nurse within one hour after inclusion, with a maximum of eight hours. Written informed consent is obtained prior to the collection of the blood samples. In case the patient is referred to the hospital and the blood samples are not collected by the GP, the study bag will be transported with the patient to the ED. Subsequently, the laboratory assistant on call will visit the patient in the hospital and collect the blood samples. Three blood tubes will be collected: 10 ml for serum, 10 ml for EDTA plasma and a 2 ml heparin tube. Lactate will be measured immediately afterwards from a single drop of blood taken from the heparin tube, using the StatStrip Xpress (Nova Biomedical) point-of-care test. The remaining blood samples will be taken to the hospital laboratory and divided into six samples of serum and six samples of EDTA plasma. The aliquots will be temporarily stored at the local laboratory at <-70°C. Two samples (1 ml serum and 1 ml EDTA plasma) will be transported to the Jeroen Bosch Hospital for CRP and PCT analyses, and the remaining samples (5x 0.5 ml serum and 5x 0.5 ml EDTA plasma) will be stored for 15 years at <-80°C at the UMC Utrecht for potential future testing.
Training and remuneration of personnel
Chauffeurs of the GP cooperatives are trained in using portable monitors for vital sign measurement and other study procedures. At the GP cooperative in Ede the chauffeurs will also be trained in the measurement of POC-lactate, as GPs will collect the venous blood samples themselves occasionally. The laboratory assistants and nurses who will be on call for the collection of the blood samples will also be trained in the POC–lactate measurement and other study procedures including the obtaining of written informed consent. Attending GPs are informed by an information letter by mail and hard copy at the GP cooperative. Leaflets with a summary of the study procedures will also be available.
Follow-up
The total follow-up time is 30 days. Patients will be asked to complete the EQ-5D-5L questionnaire22 at the end of follow-up (1) at the day of completion of the questionnaire; (2) before the onset of the recent disease episode (i.e. their health status of at least one month ago); and (3) for the worst day they remember from their recent disease episode. Furthermore, patients will be asked to report on consumption of medical resources during the 30-day follow-up period. Productivity losses are not considered, as most patients are elderly and not doing paid work. In case of no response to the questionnaire after one week, patients are contacted once by telephone as a reminder.
Data-extraction
Relevant medical information will be obtained from the patient’s (regular) GP, the GP cooperative, and the hospital. Medication use and comorbidities before inclusion are retrieved from GP electronic records, as well as information on any subsequent contacts. The medical record of the assessment at the time of inclusion is retrieved from the GP cooperative. The following data from the electronic medical record of the hospital will be collected: full reports from ED and hospital discharge; date and time of ED visit, hospital admittance and discharge (including type of ward); vital signs, EMV score, leucocyte count, thrombocyte count, creatinine, bilirubin, CRP and lactate measured in the first 72 hours after inclusion; cultures taken in the first 72 hours after inclusion; radiodiagnostic procedures in the first seven days after inclusion; antibiotic prescriptions during hospitalisation; intravenous volume therapy in the first 72 hours (defined as more than 1.5 litres of fluids in 24 hours).
Sample size
In total, 12 candidate predictors are chosen for the development of the prediction model (table 1). Using the rule of thumb of 10 events per variable,23 we need 120 patients reaching the primary outcome “sepsis within 72 hours of inclusion” in the final dataset. Prior to the start, the prevalence of sepsis based on previous research and literature was estimated to be around 12%. However, preliminary data from the patients included in the study so far, indicates the prevalence of sepsis in the study cohort to be more likely to be around 30-40%. After the first 100 cases are assessed by the expert panel, we will determine the final target sample size.
Statistical analyses
Descrriptive statistics
We will use a combination of IBM SPSS Statistics and R Statistical Software for all analyses. We will start with descriptive analyses on baseline characteristics (age, sex, comorbidities, vital sign measurements and other clinical features, blood tests results, baseline EQ-5D score), final diagnosis, hospital admission, ICU admission, length of stay, EQ-5D compared to baseline, and 30-day mortality. Results will be stratified based on whether patients do or do not meet the primary outcome sepsis.
Data cleaning
Range and distribution of all continuous variables will be graphically inspected, and any outliers (more than three standard deviations from the mean) will be discussed and dealt with accordingly. Any missing data on clinical features or blood tests, will be accounted for by applying multiple imputation techniques. Prediction model development and performance will be analysed using the imputed datasets.
Development of the prediction model
A multivariable penalized logistic regression model will be developed, based on the variables listed in table 1, for predicting the primary outcome (sepsis within 72 hours after inclusion). We will use a two-stepped approach entering and selecting clinical features first, and blood tests second. In both steps the selection of predictors will be based on a stepwise backward selection, using change in Akaike information criterion (AIC) for selecting the preferred model.24 The goal is to generate an efficient model by eliminating variables that contribute little to the model’s performance, requiring only measurement of the most important variables in clinical practice.
Continuous predictors in the model will be assessed for linear relationship with the logit of the primary outcome. Transformation of the data and splines are used if deemed appropriate based on distribution of the data.
The prediction model that has now been designed is the most accurate prediction model, by making use of continuous measurements of predictors and reflecting non-linear relationships by transformations or splines (optimal model). To make this model workable in daily clinical practice without electronic aids, a second model will be derived (clinical practice model) by categorising or stratifying predictors. Cut-offs for categorisation will be based on a combination of known and commonly used thresholds in clinical practice and optimal thresholds based on the data. This model simplification is likely to induce a performance drop with regard to the full model, which will be assessed during the analysis.
The above procedures result in the following three models: 1) Optimal model with clinical features only; 2) Optimal model with clinical features and blood tests; 3) Simplified model (with clinical features and blood tests).
Performance of the prediction model
The performance of all three models will be determined based on their discrimination and calibration. Discrimination will be evaluated based on the area under the receiver-operator characteristic (AUROC). Calibration will be assessed by plotting observed and expected probabilities and inspecting this plot graphically. Measures of calibration will include calibration slope, calibration in the large, observed/expected (O/E) ratio, and the Brier score.25 We will perform internal validation for all three models by using a bootstrap simulation. The resulting distribution will reflect optimism and the degree of overfitting.26
The SIRS criteria, NEWS score, and qSOFA score will be calculated for all individuals in the TeSD-IT study. Diagnostic performance of the existing models will be determined by calculating the same measures of discrimination and calibration as described in the sections above, and comparing these with the three models that were developed.
To assess the added value of the prediction models on top of usual care, other outcomes than sepsis will be considered. This is crucial for gaining insight in the net benefit of using the CPRs in daily practice. For example, when a patient is predicted as non-sepsis by the model, but the patient was referred by the GP, improvement compared to care as usual is only the case if hospital treatment was not needed according to the expert panel. To assess the added value, the proportion of reclassifications within the original contingency tables will be presented for the following outcomes: 1) the gut feeling of the visiting GP that “something is wrong” 2) the assessment of the visiting GP for the likelihood of sepsis, and 3) the decision of the visiting GP whether or not to refer the patient to the hospital.
Cost-effectiveness and budget impact analysis
We will measure costs from a societal perspective, including health care costs and patient costs within and outside the hospital. Productivity costs will be ignored as the average age of patients participating will exceed the age of pensioning in the Netherlands. The patient questionnaire as well as follow-up data from hospital and GP medical records is used for the calculation of total and per patient costs. The EQ-5D scores retrieved from the questionnaire are used to calculate QALYs. Our patient outcome analysis will generate QALYs for different health states that will be used in health economic modelling, such as a complicated sepsis case (including ICU admission), hospital admittance for a suspicion of sepsis, and an infectious disease episode without hospital admission. Different scenarios with different levels of implementation of point-of-care testing (POCT) for sepsis in general practice will be analysed and compared to standard of care: 100% use of the best performing testing strategy; 70%, 30% and 0% use of POCT for suspicion of sepsis (the latter representing usual care). The budget impact will be assessed using the health economic model that will be build for the economic evaluation and results will be analysed in a probabilistic way.