Validation of the Nursing Activities Score (NAS) using time-and-motion measurements in Dutch Intensive Care Units

The Nursing Activities Score (NAS) is widely used for workload measurement of Intensive Care Unit (ICU) nurses. However, the performance of the NAS to measure actual nursing time has not been comprehensively and externally validated. The aim of this study is to validate the NAS using time-and-motion measurements in Dutch ICUs. We measured nursing time for patients admitted to seven Dutch ICUs, between November 2016 and October 2017. The patient(s) that were under the care of a chosen nurse were followed by the observers during the entire shift and measurements were performed using an in-house developed web application. To validate the reliability of the NAS, we first converted NAS points per activity into minutes. Next, we compared the converted time per NAS item and the converted total nursing time per patient with the actual observed time. We used Wilcoxon signed-rank tests at nursing activity level and Pearon’s R and R 2 at patient level for these comparisons.


Introduction
There are concerns regarding excessively high nursing workload, both in general-and ICU wards 1 . Excessive high nursing workload can lead to burnout and job dissatisfaction among nurses 2 and have a deleterious effect on patients 3 . Workload has risen due to increased turnover of patients, increased complexity of patient cases together with inadequate capacity of nurses due to shortages 4 . All this makes planning of nursing capacity important. In the last 30 years different instruments have been developed to measure the nursing workload to give insight in the needed nursing staff per shift and provide much needed input for capacity planning 5 .
To assess nursing workload in the ICU Cullen et al. (1974) 6 created the Therapeutic Intervention Scoring System (TISS). The TISS was originally developed to classify nursing workload in relation to the severity of illness of ICU patients. The TISS exists of 76 therapeutic interventions that receive one to four points based on the severity of illness of the patient. It appeared that nursing workload is only partly related to the severity of illness, since less severely ill patients could also generate high nursing workload. For instance a patient recovering from a serious illness with an agitated delirium would not score high in severity of illness, but could demand very intensive nursing care, up to continuous bedside care throughout the day. This made the TISS less adequate to assess nursing workload. Therefore, the Nursing Activities Score (NAS) was developed in 2003 by Miranda et al. (2003) 7 . The NAS describes activities that largely represent the work actually performed by nurses at bedside in caring for the patients and has been developed to measure the nursing workload for each individual patient. The points assigned to the nursing activities provide an average time consumption in caring for the patients instead of representing the severity of illness. The NAS was created by using the work-sampling approach: at random moments per shift the nurse was asked what he or she was doing at that specific moment. For every activity there was a weight granted by the researchers.
The total NAS for an individual patient is the sum NAS points of all activities, varying between 0 to 177 points. A score of 100 NAS points is equivalent to the amount of care which can be provided by one Full Time Equivalent nurse during either one shift or one day. A score above 100 points indicates that the needed care can only be provided by more than one nurse 7 .
The NAS is considered a valuable tool and is widely used for workload measurement in ICUs 8,9 . However, the performance of the NAS has not been comprehensively validated.
One study showed that the NAS might either under-or overestimate the actual nursing time required by patients and therefore recommended revision of the original NAS because of inadequate measurement of nursing activities 4 .
Furthermore, research has demonstrated that the work-sampling approach, as used for the development of the NAS, does not lead to an accurate representation of the true nursing workload. This is due to the fact that the weightings of the nursing activities are based on the probability that a particular nursing activity occurred 10 . The total amount of time in a shift is divided over the occurring nursing activities. When nursing activities frequently occur or take much time, they would also occur more frequent in the work-sampling approach. However, this approach will not lead to precise measurements, but will only approximate the time of the different activities. Thus, in contrast to time-and-motion techniques in which every minute of a nursing shift is measured, the work-sampling approach does not measure the real amount of time spent on the nursing activities which could lead to less accurate results 11 . Therefore, the time-and-motion technique is considered as the best technique for time measurement 12 .
The aim of this study is to validate the NAS in the Dutch ICU setting using the time-andmotion technique, and to identify which nursing activities are under-or overestimated in the NAS.

Setting
All 82 Dutch ICUs participate in the National Intensive Care Evaluation (NICE) quality registry. Fifteen of these ICUs are participating in the newly implemented voluntary nursing capacity module 13 . Seven of these ICUs voluntarily participated in this study.
Data on characteristics of the ICUs (such as number of ICU beds) and data on patient characteristics (such as age, BMI, admission type, and mortality) were extracted from the NICE registry.

Time-and-motion
The study involved time-and-motion measurements for patients admitted to the ICU. We measured in different types of hospitals (academic-, teaching-, and non-teaching hospitals) and in different shifts (day, evening, and night). At the start of a shift one nurse was chosen by the observer. The patient(s) that were under the responsibility of this nurse were followed by the observer during the entire shift. A patient admitted for a longer time could theoretically be observed on different dates during different shifts and therefore could possibly be followed during more than one measured shift. The measurements took place during different days of ICU admission of the patients (e.g. first ICU admission day through last ICU admission day) and with different type of nurses (registered and student nurses).
Observers were researchers CM and MH and ten student nurses. The students were trained

Data analysis
Nursing activities that occurred less than ten times in the total dataset were excluded from the analysis. Most NAS items have a fixed number of NAS points but some items have different categories corresponding to different numbers of NAS points depending on the duration of that activity (e.g. bedside with hourly vital signs, bedside for two hours or more, or four hours or more). For these duration depended activities, we first used the measured time for that activity during the measurements to assign the correct number of points. For example, a nurse performed hygiene procedures on a patient for 1.2 hours during a shift according to our time measurements. This NAS item has three categories: performing hygiene procedures for less than two hours, for more than two hours, or for more than four hours. In bovementioned example, the activity took 1.2 hours and would therefore be assigned to the category for less than two hours, which corresponds to 4.1 NAS points. To validate the NAS, we first converted the originally assigned NAS points per The median and interquartile ranges (IQR) of the converted NAS times and the observed times were calculated. First, the difference between the total converted NAS times and the total observed times per patient were visualized by scatterplots. Second, the correlation between the total converted NAS times and the total observed times per patient were assessed with the Pearson's correlation test. In addition, we also assesed the R 2 , a measure for the proportion of the variance. For each nursing activity separately, medians and interquartile ranges (IQR) of the converted NAS times and observed times were calculated and differences were tested with the Wilcoxon signed-rank test. All statistical analyses were performed using R statistical software, version 3.3.2 15 .

Results
Baseline results  (Table 2). Furthermore, acute renal failure, chronic respiratory insufficiency, and cirrhosis differed between the groups, with a higher percentage in the patients in our study. For the other patient characteristics, the included patients and all Dutch ICU patients in this period were comparable.

NAS validation
Excluded nursing activities The following three NAS nursing activities occurred less than 10 times in all measurements and were therefore excluded from the analysis at activity level: activities at the pulmonary-or left atrium catheter, cardiopulmonary resuscitation (CPR), and specific interventions in the ICU (endotracheal intubation, insertion of pacemaker, cardioversion, endoscopies, emergency surgery in the previous 24 hours, gastric lavege).
Furthermore, we did not measure specifically intravenous replacement of large fluid losses and treatment of metabolic acidosis/alkalosis, since these two nursing activities are usually administered under bedside activities.

Total patient time and times per NAS item
The (R 2 =0.35) was found between the total converted NAS time and the total observed time per patient (Table 3).
For the time differences at NAS item level, we found significant differences between the converted NAS times and observed times for all items. These differences ranged from -54.6 minutes (support or care for patient or relatives for about 1 hour) to 79.  (Table 3). Since in almost each shift ICU nurses also spend time on non-nursing duties, e.g.
coaching a student or participating in an emergency team within the hospital, we performed a sensitivity analysis to determine whether these non-nursing duties were affecting the correlation. According to several studies nurses spend approximately 3 to 6% of their shift on non-nursing duties 18,19,20,21,22 . We therefore took the average of 4.5% and substracted this from the 80% of productive nursing time, which we used in this study

Conclusion
The NAS has been developed more than 15 years ago and significantly overestimates the needed nursing time for ICU patients in the current daily ICU practice. Therefore we recommend a revision of the time weights assigned to each nursing activity to get better insight in the true nursing workload and to enable a more adequate nursing capacity planning.
Additional file 2

Consent for publication
Not applicable.

Availability of data and material
The data that support the findings of this study are available from the National Intensive Care Evaluation (NICE) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of the NICE registry.

Competing interests
The authors declare that they have no competing interests.

Funding
The department of medical informatics (with C.C. Margadant, S. Brinkman, and N.F. de Keizer as employees) receive funding for data processing of the NICE registry. The funding Figure 1 The correlation between the total converted NAS time in hours and the total observed time in minutes per patient. A full nursing shift is 480 minutes. Blue diagonal shows equal converted and observed time per patient.

Supplementary Files
This is a list of supplementary files associated with the primary manuscript. Click to download.