Design
In this cohort study we used a factorial survey design in which participants were asked to evaluate clinical vignettes. Hereby importance of factors influencing decision making could be assessed. A vignette is a brief, written case history of a fictitious patient that is based on a realistic clinical situation. In the vignettes the factors of interest (in our study possible predictors of ICU outcome) were varied between the different vignettes, in which each combination was unique. We made combinations between categories of predictors, in which only one factor changed between each vignette. These vignettes were presented to the participant, residents and fellows in our ICU, in an online questionnaire using the Survey Monkey website.10 Formal approval of the institutional ethics committee was not requested, participants were informed about the anonymized use of the test results and participation was voluntary. Due to the nature of the project no sample size calculations were made.
The online questionnaire consisted of 23 clinical vignettes, in which four common ICU admission diagnoses were presented; community acquired pneumonia (CAP), pancreatitis, acute respiratory distress syndrome (ARDS) and cardiac arrest. The literature was studied for patient factors known to have impact on patient outcome in ICU for these conditions. With this information an expert panel of ICU physicians decided to add the following factors to the vignettes: age, body mass index (BMI), acute kidney injury (AKI) and presence of haematological malignancy for all diseases, and presence of chronic obstructive pulmonary disease (COPD) and severity of pneumonia measured by CURB-65 score for pneumonia, presence of chronic liver disease and severity of pancreatitis measured by Ranson score for pancreatitis, severity of ARDS and type of ARDS (distinguishing primary and secondary ARDS) for ARDS and for the cardiac arrest cases first rhythm (shockable versus non-shockable), location of arrest (in hospital versus out of hospital) and delay until start CPR.11-18 All community acquired pneumonia cases were classified as severe, according to the CURB-65 score11. In cases for ARDS and cardiac arrest all patient had a BMI of 20-25 and an age of 60-80 years old. A complete overview of all factors involved is shown in table 1.
With the complete set of factors, 324 vignettes for community acquired pneumonia (2x2x3x3x3x3), 432 vignettes for pancreatitis (2x2x3x3x4x3), 288 vignettes for ARDS (2x2x3x3x2x4) and 288 vignettes for cardiac arrest (2x2x3x3x2x2x2) were created. Completing all 1332 vignettes would be too time-consuming for participants. All unrealistic vignettes were removed and a total of 23 clinically vignettes (6 pneumonia, 7 pancreatitis, 4 ARDS and 6 cardiac arrest) were randomly selected by a team of experts/ICU clinicians based on their realistic scenarios. In each vignette only one factor changed to the next vignette. The vignettes were written by one of the investigators (LW) and discussed by the other investigators (MM and CB) on clinical accuracy and realism. The complete questionnaire can be found in Supplement 1.
For the vignettes on community acquired pneumonia and pancreatitis, participants were asked to choose between admission or no admission to the ICU, taken into account chances on survival and outcome. For all vignettes with ARDS or cardiac arrest participants were asked to estimate mortality during first 30 days of ICU stay. The options were < 40% mortality, 40-80% mortality or > 80% mortality.
Also, five questions were added on the cost of five products, which are frequently used in the ICU. Each question was multiple choice and respondents were asked to pick the right price for the product.
Demographics of the participants were collected at the start of the questionnaire. Recorded variables included working experience in ICU care, previous experience in residency or fellowship and primary postgraduate medical education programs. The first questionnaire was carried out in January 2018, the second, after the educational intervention, in April 2018.
The questionnaire before and after the educational intervention was similar.
Setting
This study was executed at the Intensive care department of the Amsterdam University Medical Center, location Academic Medical Center, at the University of Amsterdam. The department is a 34-bed mixed medical-surgical ICU, where residents of various postgraduate medical education programs are trained (e.g. internal medicine, anaesthesiology, surgery, neurosurgery, cardiology, emergency medicine). In addition, yearly 7 fellows are trained to become an intensivist as a subspecialty of their training in anaesthesiology, internal medicine, cardiology or neurology.
Participants
The study population consisted of ICU residents with medical training in anaesthesiology, internal medicine, emergency medicine, neurosurgery, cardiothoracic surgery or general surgery, and fellows in Intensive Care Medicine, with a medical specialization in internal medicine, anaesthesiology or neurology. All physicians were employed on the ICU of the Amsterdam Medical Centre, a tertiary clinic, during the study period.
Intervention
Between January and April 2018 an educational program on outcome and cost of ICU treatment was implemented in the regular educational program. This program consisted of flipped classroom sessions on outcomes of patients admitted to the ICU with a certain illness, comorbidity or patient characteristic; lectures on costs if ICU care and cost reduction and organization of ICU and a weekly quiz.19 The flipped classroom sessions focused on COPD, haematological malignancies, pancreatitis, community acquired pneumonia, cardiac arrest, age and BMI. In total eight different flipped classroom sessions were organized. The classical lectures focused on organization of ICU care, assessing the outcomes of ICU care, cost reduction on laboratory measurements and financing ICU care. Since all of our physicians work on irregular shifts all lectures were held twice and handouts of the lectures and acquired information of the flipped classroom sessions were shared between all physicians.
The weekly quiz with a fictional but realistic case and a question on the actual costs of a certain investigation, medication or material was sent to all participants every Monday. After the first week a ranking scale was made with all participating competitors and updated every week.
Potential effect modifiers, confounders and bias
Potential modifiers of the measured effect of the intervention are previous experience in ICU care, outcome of ICU care, cost and already gathered knowledge on high-value cost-conscious care during general medical training or residency. For this reason, we included all residents and fellows employed at our ICU during the intervention and asked about their level of experience in ICU care. There was risk of inclusion bias, since participation was voluntary and this could select only participants interested in the subject of the study. To prevent selection bias we briefed all residents and fellows about the study and potential of high-value cost-conscious care education to gain interest in the study. We tried to minimize loss of follow up by asking to complete the final questionnaire on several occasions.
Statistical analysis
Descriptive statistics were used to describe the characteristics of the participating residents. We used logistic regression to determine the first outcome, relative importance of the factors for ICU admission, in which ICU admission was set as determinant and the factors as independent variables. Data are shown as odds ratios (OR) with 95% confidence interval.
For the second outcome, the importance of the factors for the estimate of mortality, we used univariate multi-nominal regressing, with percentage of survival as determinant and the factors as independent variables. Data are shown as odds ratios (OR) with 95% confidence interval.
The third outcome, the estimation of costs of products used regularly in ICU, was noted a percentage different from the true cost. Results are shown as means with standard error of the mean (SEM). A students t-test was performed to compare answers before and after education.
IBM Spss version 25 was used for the statistical analyses and a p value of < 0.05 was considered statistically significant.