The Neonatal Withdrawal Assessment Tool (NWAT): Pilot inter-rater reliability and content validity

27 OBJECTIVE: There is no validated tool to assess withdrawal in preterm infants in the newborn 28 intensive care unit (NICU) being weaned from pharmacological sedation. 29 STUDY DESIGN: The Neonatal Withdrawal Assessment Tool (NWAT) was developed to 30 address this gap in clinical practice. In this pilot study, the NWAT was assessed for inter-rater 31 reliability (IRR) and content validity. 32 RESULT: Fifty-one NICU providers scored two standardized simulated cases, then 20 paired 33 provider assessments were completed on 5 preterm infants. The overall IRR was 95.6% on the 34 simulated cases, and 98.8% on the 5 pilot infants. A provider survey assessed for content 35 validity; more than 85% of provider participants strongly agreed/agreed that the NWAT 36 adequately measures withdrawal in critically ill infants. 37 CONCLUSION: The NWAT demonstrated high IRR and content validity for assessment of 38 withdrawal in preterm infants. Further studies in a larger more diverse patient population are 39 needed before wider adoption into clinical practice. 40


INTRODUCTION
Preterm infants in the neonatal intensive care unit (NICU) often require pharmacological treatment with sedating medications such as opioids and benzodiazepines due to severe clinical illness, resulting in subsequent iatrogenic withdrawal.Sedation can prevent agitation-related adverse outcomes including risk for accidental extubation and increased oxygen consumption, and reduces infant stress 1 .However, exposure to opioid and other sedating medications in preterm infants has been associated with risk for neuroanatomical changes as well as motor, behavioral and cognitive changes in the first two years of life [2][3][4][5] .This suggests that the length of exposure and the cumulative doses of exposure to opioids and benzodiazepines be ideally minimized while maintaining appropriate care for the critically ill preterm neonate.
In addition to the scenario of a preterm infant who receives sedating medications postnatally for critical illness, infants with in-utero opioid exposure who are born preterm often require treatment for neonatal opioid withdrawal syndrome (NOWS) with opioid medications [6][7][8][9] .
For both this scenario and the iatrogenic withdrawal scenario described above, there is a paucity of research in how to best assess withdrawal in the preterm infant, providing minimal guidance on how to best wean pharmacological sedation in this population 6 .Currently, there is no validated, gold standard assessment tool specifically designed for this common scenario in the NICU.
For preterm infants, there are validated measures for the evaluation of pain, including the Neonatal Pain, Agitation and Sedation Scale (N-PASS), however this is targeted to assess acute pain and was not designed to assess withdrawal measures in the setting of prolonged pharmacological sedation [10][11][12][13] .The Withdrawal Assessment Tool -Version 1 (WAT-1) metric is the best tool currently validated for diagnosis and assessment of iatrogenic withdrawal in older pediatric patients with meta-analysis data demonstrating high reliability and validity [14][15][16] .
However, this tool was not created for nor assessed in the preterm infant and therefore not ideal for use in this population given significant differences in the presentation of withdrawal in preterm infants compared to older infants and children.
Withdrawal assessment tools for NOWS after in-utero opioid exposure in full-term infants are commonly utilized in clinical care.The Finnegan scoring tool and modifications of it have been the standard of care since the 1970's, developed for use in full-term non-critically ill infants who were exposed to opioid in-utero 17 .One study found that in comparison to term infants, NOWS assessments using the traditional Finnegan scoring tool in a cohort of preterm infants resulted in less frequent scoring for sleep disturbance, tremors, muscle tone, sweating, nasal stuffiness and loose stools and more frequent scoring for hyperactive moro reflex, tachypnea, and poor feeding 8 .Another study that utilized Finnegan scoring to assess NOWS found that prematurity reduces the severity of the NOWS, though it could be a reflection of inadequate assessment of preterm infants by the Finnegan tool given it was validated for use in full-term infants 7 .The Finnegan tool was shown to have poor psychometric properties in recent studies and its utilization has decreased over the past five years 18 .Newer NOWS assessment methods such as the function-based Eat, Sleep, Console (ESC) approach and assessment tool were developed for use in full-term infants cared for in non-intensive care settings, thus their utility are questionable in sick preterm infants in the NICU who often require respiratory support, feeding support, and are unable to receive the same consolability measures 19,20 .
A specific assessment tool designed to appropriately assess withdrawal in critically ill preterm infants and preterm infants with in-utero exposure is particularly needed to guide weaning from pharmacologic treatment.Here we describe the development and preliminary inter-rater reliability (IRR) and content validity of a modified WAT-1 scoring system, the Neonatal Withdrawal Assessment Tool (NWAT), designed to assess appropriateness of weaning sedative medications in the preterm neonate.

Tool Development and Description
First, a literature search was conducted for pain and sedation assessment tools suitable for use in the neonatal population, focusing on the preterm population.Then, Boston area NICUs were contacted to determine which tools were in clinical practice in our geographic region.The most common tools utilized were the WAT-1, N-PASS, and Finnegan scale for infants weaning from long-term sedation 14,17,21 .The findings of the literature search and copy of existing tools were then shared with nursing leadership and bedside nurses in the NICU, with the WAT-1 being identified as the most congruent with goals of care during the weaning phase.Given the differences in withdrawal manifestations in preterm infants, the following modifications were made to the original WAT-1 to create the NWAT tool.These modifications were based on NICU nursing feedback on caring for preterm infants undergoing withdrawal.The following items were removed from the WAT-1 entirely due to low likelihood of seeing these items in a preterm neonate based on previously published descriptive reports: 1) Any sweating, 2) Uncoordinated / repetitive movements, 3) Yawning or sneezing. 8The following items were modified to better reflect definitions of symptoms seen in infants: 1) 'Any vomiting / wretching / gagging' was changed to 'feeding intolerance due to withdrawal' given that most preterm infants are on nasogastric feedings and not manifesting emesis, wretching, or gagging, and 2) 'Temperature >37.8' has been changed to 'temperature instability, tachycardia, or tachypnea due to withdrawal' to be more consistent with vital sign changes seen in preterm infants with withdrawal 8,22,23 .The remaining items were not modified from the original WAT-1 scale.
The final 8-item NWAT consists of 3 items related with the infant's medical record for the past 4 hours, 2 items on pre-stimulus observation for two minutes, 2 items on stimulus observation during routine care, and 1 item on post-stimulus recovery (Figure 1).The data abstracted from the medical records includes (1) any loose/watery stool, (2) any feeding intolerance as vomiting, wretching, gagging, (3) temperature greater than 37.8 in an open crib or temperate instability in isolette, heart rate or respiratory rate greater than the infant's baseline.The pre-stimulation items include 1) state, and 2) tremor.The stimulation observation includes items: 1) startle to touch, and 2) muscle tone.Finally, the post-stimulus recovery contains

Inter-rater reliability
The inter-rater reliability (IRR) was established by a stepwise approach.All nurses and physicians working in the NICU who trained on the NWAT by the gold standard raters (EW and CS) were encouraged to performed assessments of two standardized simulated case examples according to NWAT.The recorded scores were compared with the scores given by the two gold standard raters who initially created and assessed the cases.
After the NWAT was fully implemented within the NICU, two gold standard raters independently scored 5 preterm infants weaning from prolonged sedation at the same time as the primary nurse on a total of 20 occasions.

Content validity
The content validity of the NWAT was examined by administering an optional anonymous survey to all NICU staff following NWAT training.The survey included 10 questions on a 5-point scale rating the relevance of the content.The scale went from "strongly disagree" to "strongly agree" according to the Likert scale 24 .

Data Analysis
This study was approved by the Boston University Medical Campus Institutional Review Board as an exempt quality improvement study with waived informed consent.All statistical analysis was performed in R version 4.1.1(2021-08-10).First, the IRR was assessed by determining the percent agreement on each item of the NWAT on the two simulated cases, as well as the mean (SD) and median (IQR) for the total score, and percent agreement on the treatment plan for the infant (wean or not wean) based on the total score.Each case was assessed separately, and then the total IRR for the 2 standardized cases determined based on the 8 items in the scale for all provider responses.The same analysis for determination of IRR was then conducted for the 20 assessments on the 5 pilot infants.
Next, content validity was assessed by determining the number and percentage of survey responses for each item in which there was an "agree" or "strongly agree" response, versus other responses (neither agree or disagree, disagree, or strongly disagree) to the survey about the content of the NWAT tool.

RESULTS
The quiz containing the two standardized simulated cases was sent to all 10 NICU attendings and 60 NICU nurses who work at our center.A total of 51 providers responded, a 73% response rate.
The percentage of correct scores for each item ranged between 66.7-100% for case 1 and 96.1-100% for case 2. Six of the items were scored >94% correctly in both cases.Among the 8 items, "temperature instability, tachycardia, or tachypnea due to withdrawal" had the lowest interrater reliability (66.7%).All total scores were within 1 point of the correct score in both cases.Overall, 100% of the participants made the same decision for sedation weaning in both cases, congruent with the gold standard raters.The overall IRR was 95.6% for the simulated cases (Table 1).
The NWAT was then piloted on 5 infants in the NICU undergoing sedation weaning using the new protocol.These infants did not have in-utero opioid exposure.The mean gestational age at birth was 27.5 weeks (range: 23.2 to 29.4 weeks) and the mean birth weight was 944 grams (range: 480-1510 grams).Infants were managed with opioids and benzodiazepines for sedation.Clinical characteristics of the 5 infants is shown in Table 2.These 5 infants underwent a total of 20 withdrawal assessments by two gold standard raters in addition to being scored by the bedside nurse.The IRR of 8-scoring items ranged between 95% and 100%.The decision for sedation weaning demonstrated 100% agreement between the bedside staff and gold standard raters.The overall IRR was 98.9% (Table 3).
Lastly, all NICU nurses and physicians (n=70) were asked to complete the anonymous survey.There were 21 individuals who completed the survey (30% response rate).The mean number of years working in a NICU setting was 7.9 (SD 9.9) years.More than 85% of participants strongly agreed/agreed that the NWAT adequately measures withdrawal from opioids and benzodiazepines in critically ill infants in the NICU giving a sufficient cutoff criteria for weaning.
More than 90% of participants strongly agreed/agreed that the information needs to be gathered from the medical record and at the observation periods (pre-stimulus, during stimulus, poststimulus periods) were adequately described on the NWAT.The NWAT was felt to be a better/more specific tool compared with Finnegan Neonatal Abstinence Syndrome Assessment Tool (FNAST) and ESC NOWS Care Tool for critically ill neonates in the NICU weaning from sedation.The survey results are shown in Table 4.

DISCUSSION
This study describes the preliminary IRR and content validity of the NWAT which has been developed to quantify the withdrawal signs to help guide providers in their decision making of weaning sedative medications in preterm infants at risk for iatrogenic and/or in-utero withdrawal.
The NWAT demonstrated high IRR of >95% and high content validity for assessment of withdrawal signs in both simulated cases and preterm infants in the NICU.
Overall, IRR was very high above the generally accepted cut-off of 80%.However, "temperature instability, tachycardia, or tachypnea due to withdrawal" had the lowest IRR in the standardized cases.These vital sign changes could be interpreted as related to prematurity which might cause lower interrater reliability compared to the other items.Heart rate has been demonstrated to be a strong marker of pain and stress in preterm infants, and with perhaps a need for further clarification in tool description of when to indicate a particular score for this time 25,26   .The lowest IRR was for "startle to touch" in the five pilot infants, whereas there is strong IRR in the two standardized cases.This could indicate that interpretation of startle response may remain provider dependent despite education material.Differences are known to exist in the preterm population by gestational age for the Moro reflex and other motor movements 25,27,28 .This may also indicate a need to better define this item in subsequent versions of the NWAT tool description in subsequent versions of the tool.
Strengths of our study include that it is the first description of a withdrawal assessment tool specifically created for use in preterm infants.The tool was demonstrated to be both highly reliable in simulated and actual patient settings, with high content validity amongst NICU physicians and nurses.The NWAT was also demonstrated to be highly feasible, easily utilized and well received by nursing staff, and rapidly adapted and implemented into clinical care protocols.
However, this study does have some limitations.First, the response rate for the anonymous optional survey was only 30% and reflected primarily staff who had been working in the NICU for at least 5 years, which could reflect bias in those who responded to the survey compared to those who chose not to respond.The response rate for the IRR quiz was higher as this was a recommended training for all staff.We were also limited in the small number of infants assessed with the NWAT; we plan to examine the NWAT in a larger number of infants and to assess validity further including impact on infant outcomes in future studies that are adequately powered to look at these clinical outcomes.
Future directions for further validation the NWAT tool include trialing use of the tool in other care settings such as larger NICUs, particularly among full-term infants with iatrogenic withdrawal in comparison with other tools such as the WAT-1 or FNAST.In addition, assessing the use of the NWAT tool specifically in a cohort of infants with in-utero opioid exposure who are born preterm is warranted in future studies, with or without comparison to the FNAST or ESC NOWS assessment method.Outcomes to be evaluated include length of sedation medication weans, length of hospitalizations, central line days, adverse events and complications, long-term outcomes, and further assessments of content validity from a larger NICU provider base including providers with more varying years of clinical experience 14,15,29,30 .
Given the ease of adaptation and lack of current alternative standard of care, the NWAT could rapidly become the new standard of care for preterm infants in the NICU weaning from sedation as well as preterm infants with NOWS.If further validated in future studies, it could have a significant impact on neonatal care, including the potential to shorten pharmacologic treatment courses and decreasing exposure to potentially harmful medications, while also optimizing infant comfort and minimizing excessive signs and symptoms of withdrawal [31][32][33] .This in turn could result in improved neurodevelopmental outcomes for this high-risk cohort of preterm infants.
In conclusion, this is the first step in developing a tailored tool for preterm infants with iatrogenic withdrawal or withdrawal due to in-utero opioid exposure leading to potentially better practice and improved neonatal outcomes.

CONFLICTS OF INTEREST:
The authors declare that they have no competing interests.

ETHICS APPROVAL:
This study was approved by the Institutional Review Board of the Boston University Medical Campus as an exempt protocol with waived informed consent.

Supplementary Files
This is a list of supplementary les associated with this preprint.Click to download. SupplementalFigure.pptx Figure).

TABLES 383 Table 1 . Inter-rater reliability from two standardized case quizzes from 51 providers 384
The correct score was determined by the two gold standard raters who designed the two case quiz. *