Development and validation of a risk assessment scale for hepatic encephalopathy in cirrhotic patients

Abstract

Background: This study aims to develop a risk assessment scale to assess and quantify risk of hepatic encephalopathy in cirrhotic patients.

Methods: Data were derived from two Delphi rounds and a cross-sectional survey of 276 cirrhotic patients. The minimal consensus were a mean score 3.5 on a scale of 1 to 5, and a coefficient of variation 0.25. The reliability and validity of the scale were validated by a cross-sectional survey.

Results: After two Delphi rounds and item analysis, a final 23-item scale covered two dimensions. The total Cronbach’s αretest Cronbach’s α, interrater Cronbach’s α and split-half Cronbach’s α were higher than 0.7, showing an acceptable reliability. The AUC was 0.792, and the sensitivity and specificity were 83.9% and 59.6%, respectively, indicating that the scale had a good predictive validity.

Conclusion: This scale may be a promising tool for estimating and ranking risk of hepatic encephalopathy. Future research is necessary to apply it in hepatic encephalopathy.

background

Hepatic encephalopathy, which has high morbidity and mortality in cirrhotic patients, usually occurs in patients with decompensated cirrhosis [1]. An increasing number of studies show that hepatic encephalopathy is associated with poor prognosis, increased duration and cost of hospitalization, and a high probability of readmission [2], as well as poor outcomes, including traffic accidents, falls and fractures caused by motor impairment [3]. Consequently, measures to monitor and identify cirrhoticapatients with elevated risk would have a critical impact. Type C hepatic encephalopathy is common in cirrhotic patients. Elimination of precipitating factors plays an important role in preventing hepatic encephalopathy. Most importantly, identifying the causes and quantifying risk levels are the foundational countermeasures against hepatic encephalopathy and Minimal hepatic encephalopathy.

The consensus on hepatic encephalopathy identify the potential causes as serious liver disease, abnormal portal-systemic shunts and metabolic disturbance without chronic liver disease, as well as precipitating factors [4]. Expect for the Psychometric Hepatic Encephalopathy Score (PHES) and Critical flicker frequency test [5], various types of professionally used models composed only of indicators based on the outcome of liver or kidney function have been designed for prognosticating mortality and complications relating to hepatic encephalopathy [6]. However, little focus has been placed on precipitating factors, such as transjugular intrahepatic portosystemic shunt (TIPS), infection and gastrointestinal bleeding.

A previous review suggested that the causes and precipitating factors could also be classified into six categories, including increasing ammonia production or entry into the brain, portal-systemic shunts, vascular occlusion, primary hepatocellular carcinoma and drugs [7] Moreover, comorbidities could increase the risk of hepatic encephalopathy, such as diabetes, sarcopenia, and intestinal bacterial overgrowth [8-10]. Interestingly, poor adherence to lactulose treatment might result in recurrent hepatic encephalopathy [11].

The model by Riggio [12] and the risk score by Tapper [13] included factors for liver function and factors for precipitating factors; but one was designed for OHE, and the other one excluded inpatients. Nurses can use psychometric instruments such as the number connection test, the digit-symbol substitution test, and the PHES to assess covert hepatic encephalopathy. However, between assessment or administration and feedback, these nurse-conducted procedures failed to analyse risk factors. Consequently, in addition to the tools, scales or questionnaires adopted by the medical staff, data on factors related to cirrhosis must be gathered separately. Therefore, existing hepatic encephalopathy risk assessments might add to the time demands and workloads of nurses as the nursing process develops because of the complexity of the procedures and the high economic cost of these tools.

Can this added burden on nurses be avoided? It is clear that past practices did not solve the problems that nurses faced. There is an urgent need for a tool that addresses precipitating factors and can be administered. The factors mentioned above could be used to develop a tool for assessing the risk of hepatic encephalopathy. Thus, the study assumed that there is an instrument that systematically focuses on assessing and managing liver function and precipitating factors to estimate the risk for hepatic encephalopathy in liver cirrhosis. To promote quality preventive care and decrease the risk of hepatic encephalopathy, this study aims to develop a practical risk assessment scale for hepatic encephalopathy in cirrhotic patients that could serve as an evaluation tool to deliver good primary or secondary care.

Methods

Two cross-sectional study designs was used for analyse the correlations between factors, risk of hepatic encephalopathy and risk assessment scale by two Delphi round and a cross-sectional survey. The study design consists of the following two phases and 4 steps: first phase, a modified Delphi technique was used to gather experts’ opinions on the content of the risk assessment scale for hepatic encephalopathy in cirrhosis, blindly and iteratively; and second phase included construction of risk assessment scale and validation.

Phase Ⅰ: the Delphi study

The literature was searched using PubMed, Web of Science and other relevant databases. All relevant keyword variations were used for terms related to predictors of hepatic encephalopathy, and the data reviewed from the literature and the retrospective study were considered as items for the initial framework. Then, the initial draft of factor statements was modified through a round-table discussion, and according to these results, the Delphi technique was used to help develop risk statements and identify important factors.

The Delphi technique was designed to obtain expert consensus by gathering two rounds of in-depth information in a blinded manner from April to July in 2018, until the opinions of the experts reach unanimity. One who defined as supervisors in senior positions with at least ten years of full-time experience in the field [14]. However, anyone whose questionnaire was incomplete was excluded. Accordingly, 27 experts who had published work on cirrhosis were invited by email and gave informed consent for round one, and 26 experts provided their input. In the same way, 26 experts participated in round two, of whom 23 returned the survey. The sample sizes for the two Delphi surveys were sufficient to validate a useful amount of feedback and achieve consensus on the list of items [15].

Every Delphi round began with a normal email that all experts were requested to answer within two weeks; if a participant failed to submit the questionnaire on time, reminders would be sent. The second Delphi round began after the data from the first round was analysed. After two rounds of consultation, the experts reached a satisfactory level of agreement, and then the process ended. Experts were asked to score the importance of each indicator on a scale from 5 points (“should definitely be most important”) to 1 point (“should definitely be least important”) [16]. The criteria defining a minimal consensus were a mean >3.50 and a coefficient of variation (CV) <0.25 in both rounds [17]. A senior consensus was considered to occur when an item was scored 4 or 5 by 80% of participants [18]. Items that did not achieve a minimum consensus were removed from the scale. If an item achieved a minimal but not a senior consensus, our group would discuss the item and then modified or removed it. Finally, new items were added or regrouped, mainly based on the experts’ proposals.

Phase Ⅱ: cross-section study

A cross-sectional investigation with cluster sampling was carried out from August to September in 2018 in the department of infectious diseases, the department of digestion, and the department of hepatobiliary surgery in a third-class nursing hospital. The required sample size was at least five times the number of items on the scale. The inclusion criteria for the cirrhotic patients were age older than 18 years and having been diagnosed via symptoms, previous tests, liver biopsy, etc. Patients with mental illness, metabolic encephalopathy, intracranial lesions and toxic encephalopathy were excluded, as were patients who did not provide informed consent.

The critical ration, correlation, exploratory factor analysis and internal consistency were used for item analysis. The initial value assignments of the risk factors were 0 and 1. If a factor had more than two levels, they would be numbered in order (0, 1, 2, 3, …). Factors with more than two levels include serum albumin, serum prealbumin, total bilirubin, and prothrombin activity. The total scores of participants were examined in ascending order. And the participants at before 27% of total were divided into lower group, and the participants at after 73% of total were divided into higher group; each item was compared between these two groups. Item analysis was also conducted by correlating the item scores with the total scores. Correlations between item scores and total scores were acceptable only if statistical testing showed that they were significantly different from zero. The study asked for the factor load was higher than 0.4 and communality of factor was higher than 0.2 by exploratory factor analysis. If an item was not achieved, it was considered be removed. After that, factor analysis was used to determine the factor weights for the sake of objectivity [19].

In addition, some of them who were selected by a random number table were measured by the same examiner two weeks later, result in analysing retest reliability. Moreover, patients were also evaluated simultaneously by two nurses who underwent the same training, leading to analyse the inter-rater reliability. And split-half reliability was tested by odd term and even term. After total scores were summed according to the grading scale, item analysis and factor analysis were repeated. The area of ROC was higher than 0.7, showing a good predictive validity. The Child-Pugh score was used to analyse the criterion validity.

results

General information on the experts

Twenty-seven experts were contacted, of whom 26 (response rate, 96.3%) accepted and completed the first round survey and 23 (response rate, 88.5%) completed the second round (Table 1). The response rate for each round exceeded the recommended minimum of 70%. The experts’ authority coefficients for the first and second rounds were 0.928 and 0.932 respectively, reflecting good content validity.

Result from the first round

Six dimensions reached both types of consensus, but two dimensions did not. The mean and CV ranges of 44 potential factors were 2.8~4.6 and 0.104~0.294, respectively. Among 44 potential indicators, 12 (27.3%) failed to achieve a mean >3.5 and 9 (20.4%) failed to achieve a CV <0.25. In response to these results, 12 indicators were removed. Kendall’s coefficient of concordance (Kendall’s W =0.405, 0.383, P<0.001) for the group agreement showed significant but not strong agreement among dimensions and items (Table 2). On the one hand, it is supposed that portal-systemic shunts should be divided into TIPS, surgical portal-systemic shunts and spontaneous portosystemic shunts according to the cause of the shunt. Experts also suggested new items, including course, prealbumin, indocyanine green test, multiple gastrointestinal bleeding history, biliary tract infection, intestinal bacterial overgrowth, hepatopulmonary syndrome and hepatorenal syndrome. In addition, nine items were merged into four items: 1) previous TIPS and current TIPS, 2) porta-caval shunt, spleno-renal shunt and mesenteric-caval shunt, 3) excessive intraoperative time and excessive intraoperative or postoperative bleeding; and 4) pulmonary infection and urinary tract infection. A total of 35 new items proposed by experts were added to the second round.

Result from the second round

Eight dimensions all achieve a mean>3.5 and a CV <0.25. Among 35 indicators, five factors were removed for failing to achieve a mean >3.5, although all factors achieved a CV <0.25 (Table 2). Thus, five factors were removed. Furthermore, Kendall’s coefficient of concordance (Kendall’s W =0.520, 0.441, P<0.001) for the group agreement indicated a highly significant level of group consensus. In addition, esophageal and gastric varicose bleeding and other types of gastrointestinal bleeding were merged into one item. Ultimately, 29 items, including liver function (6 items), portal-systemic shunts (3 items), gastrointestinal bleeding (2 items), electrolyte disorder (5 items), infection (4 items), drugs and diet (3 items), operative factors (1 item), and comorbidities (5 items), were retained after group discussion.

Pilot research

The researcher took a mean of 10 min to complete initial questionnaire for 50 participants, regarding the judgement of items and the evaluation of clarity. Most items had not less than one case, except surgical portal bypass, benzodiazepine drugs and hepatorenal syndrome. And the total Cronbach’s alpha was 0.739, suggesting a good consistency.

Item analysis

A total of 276 patients (202 male, 74 female; mean age 53.66±11.52 years) with cirrhosis enrolled, and hepatic encephalopathy was present in 31 cases (11.2%). The most frequent etiology of cirrhosis in the study was virus (81.9%), alcohol (9.4%), virus and alcohol (2.2%), and other etiologies (6.5%), including autoimmunity and no obvious causes. The total scores of 276 cases were examined in ascending order, and the bottom 96 participants (lower group, total score £4) and top 91 participants (higher group, total score ³10) were identified. Each item was compared between these two groups, so 7 items could be deleted. Correlations between item scores and total scores were acceptable only if statistical testing showed that they were significantly different from zero, so 7 items could be deleted. Two items had double factor load and the factor load of four items was lower than 0.4. The value of 20 items of correlation of correction terms was lower than 0.3 and the cronbach’s alpha of 7 items was not lower than total cronbach’s alpha. As a result of the item analysis, the surgical portal-systemic shunts item was merged with the TIPS item. Spontaneous portosystemic shunts, operative factors, multiple gastrointestinal bleeding history, benzodiazepine used and hypovolemia, were removed, and 23 items were retained eventually.

Factor analysis

This study likewise found that the Kaiser-Meyer-Olkin index (KMO = 0.756) and Bartlett’s Sphericity tests (X2 = 1274.15;P < 0.001), meaning that the data were suitable for factor analysis. All items’ factor load were higher than 0.4. The cumulative variance contribution rate of each dimension was higher than 40%, suggesting a good structure validity. Twenty-three items on the hepatic encephalopathy risk assessment scale generally had especially large effects. The range of possible total scores on the scale was from 0 to 76 points, but the maximum total score obtained for the survey in this study was 42 points.

Reliability analysis

A minimum consensus in the early stage of scale development was defined as Cronbach’s alpha ≥0.7, suggesting good reliability for the scale. The total Cronbach’s alpha for all items on the scale was 0.774, reflecting an acceptable degree of internal consistency reliability. Moreover, there were statistically significant degrees of test-retest reliability and inter-rater reliability (r=0.863, 0.910, P<.001), showing that this scale had good consistency. The split-half reliability, calculated from the total scores of odd and even items, was 0.834 (P<0.001, Spearman’s correlation).

Validity analysis

All dimensions, except portal-systemic shunts was significantly correlated with total scores (r=0.167~0.742, P<0.05, Pearson’s correlation). What’s more, the correlation of the scale to the Child-Pugh score was 0.500, reflecting a moderate correlation. The scale had an area under the ROC curve (AUC) of 0.792 (95% confidence interval, 0.762~0.862, P<0.001), and the cut-off score was 17.5. The sensitivity and specificity of the scale were 83.9% and 59.6%, implying an acceptable degree of predictive validity. When the cut-off score was 17.5, the scale could effectively differentiate participants with hepatic encephalopathy from participants without hepatic encephalopathy (Z=-5.302, P<0.001), illustrating a better discriminant validity.

discussion

The study developed a final risk assessment scale for hepatic encephalopathy by Delphi technique and clinical practice. And this research has proved the scale has good composite structure, showing a good reliability and validity. Since nurses mainly pay attention to precipitating factors to decrease the risk of hepatic encephalopathy, this method associated with precipitating factors might be suitable for nurses and provides the benefits of averting subjective bias or omission. So the scale will be an implement to healthcare providers to evaluate the risk for hepatic encephalopathy in cirrhotic patients, and manage and offer more educational strategies for these patients. The original study suggests the liver function is the foremost part and precipitating factor could enhance predictive power of this scale.

In the Delphi technique, a suitable number of representative experts in the relevant specialty is consulted, and items are removed or incorporated for one is excrescent or unsuitable to estimate the risk of hepatic encephalopathy [15]. This method also can add new items like course of cirrhosis and hepatorenal syndrome, and define intimate judgment like TIPS that is during the one year or ascetic. Although a few items could not reach a percentage of 80%, all items achieves a mean>3.5 and CV<0.25. Moreover, the percentage of a mean>3.5, CV<0.25, consequence>80% and Kendall’s W in second round are better than the first round, providing further evidence of good agreement [17, 18]. This scale is unique in that specific scores for all items were defined in the final stage, including precipitating factors and not merely predictors based on the outcome of liver or kidney function, like albumin and total bilirubin.

The study found most items are suitable for assessing the risk of hepatic encephalopathy in cirrhotic patients by pilot study. Some items are removed by items analysis for increasing stability and dependability. Combining items analysis with delphi technique synchronously achieves subjectivity and objective viewpoints, and avoids some important items omitted. In addition, total Cronbach’s α, test-retest reliability, inter-rater reliability showed that this scale had good internal and external consistency [20]. The results from item analysis show that this scale has two dimensions including liver function and precipitating factors, and the Cronbach’s α of each dimension are more than 0.7. The predictive validity of our scale was acceptable (AUC=0.792), each dimension also is not less than 0.7. And as its critical value score was 17.5, this scale could availably distinguish patients with hepatic encephalopathy from patients without hepatic encephalopathy (P<0.001). That is, this scale might be useful to a certain extent in assessing cirrhotic patients during their hospital stays.

Here, one of the main problems associated with hepatic encephalopathy was one that stresses the patients’ precipitating factors to perform their usual or individualized preventive activities. Previous researches [12, 13, 21] suggested that there are four instruments based on the clinical factors to predict overt hepatic encephalopathy or cover hepatic encephalopathy. Compared with those tools, items of liver function like albumin, bilirubin and previous hepatic encephalopathy are similar to scale. However, this scale covers many precipitating factors, which is different from those predicted tools essentially. Two predicted tools developed by Tapper (2018) and Lable (2019) considered PPI and ascites, reflecting that the precipitating factors also are important for hepatic encephalopathy.

Varying under different conditions, risk levels are unstable and individualised care depending on risk levels of hepatic encephalopathy [22]. Once a patient with cirrhosis develops overt hepatic encephalopathy, the patient is easily predicted to have covert hepatic encephalopathy. It shows the consequence of history of hepatic encephalopathy to evaluate risk levels for hepatic encephalopathy, and precipitating factors furthermore boost to develop hepatic encephalopathy. Precaution of hepatic encephalopathy includes avoiding the possible precipitating factors [22]. Disease, treatment and care measures also have effects on the precipitating factors, so evaluator must consider the potential situations that may arise in the process of treatment and nursing. Because nurses’ clinical thought and nursing practice often are based on precipitating factors and observation of liver function to give most benefit to prevent from hepatic encephalopathy. Thus, this scale considers these reasons and combines quantification, management and preventive measures to meet different individual needs, nurses could set warning signs to increase assessment frequency according to risk levels, and doctors could give lactulose, probiotics, rifaximin or antibiotic in terms of symptoms at different phases, adding more time to communicate with patients and caregivers. A published study showed that a consistent quality improvement could reduce the length of hospitalization and the 30-day readmission rate for cirrhotic patients [23]. Hence, the risk scale might be useful as an instrument to help establishing consistent quality improvement for nursing plans or feed-forward control by frequent evaluation.

Since the scale encompasses most recognizable risk factors for hepatic encephalopathy, some relevant educational materials could be designed to help the patients acquire relevant preventive knowledge and adopt methods for early prevention [24]. As a result, the patients who are at risk may benefit from more frequent support and education of caregivers and may learn to prevent falls and recurrence with the help of precise advice [25]. After repeated testing and improvement, full implementation of the scale could conveniently assess hepatic encephalopathy risk levels of every cirrhotic patient and help to develop nurses’ subjective initiative, critical thinking and quality of care services, even converting them to an active prevention model.

There were some limitations to this study. Time limitations restricted the number of patients we only researched at one hospital, causing representativeness of sample partial. In addition, it is supposed that the scale could measure MHE; however, there were no measures of MHE at the time of the study and information is lacking on the connection between scale and MHE. Potential bias arises from the use of various models of treatment among three departments, manifested as differences in areas such as methods of operation and nutritional support.

conclusions

The Delphi technique and a cross-sectional study were mainly performed to develop a risk assessment scale for hepatic encephalopathy in patients with cirrhosis. And the validation study confirmed that the items on the scale were feasible to use in a professional setting. Moreover, the reliability and validity of the scale were acceptable. Therefore, the scale could be considered an available tool for monitoring risk of hepatic encephalopathy in cirrhotic patients. And, implementing this scale could provide comprehensive and beneficial support to prevent hepatic encephalopathy.

declarations

Abbreviations

TIPS: transjugular intrahepatic portosystemic shunt; KMO: Kaiser-Meyer-Olkin; AUC: area under the ROC curve.

Acknowledgement

We were grateful to all experts who participated in Delphi panel. The authors would like to thank the General Program of Medical Research Project of Chongqing municipality for funding this study.

Funding

This project was supported by the General Program of Medical Research Project of Chongqing municipality (2017MSXM032). The content is solely the responsibility of the authors and does not necessarily represent the official views of Medical Research Project of Chongqing municipality.

Availability of data and materials

Datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Authors’ contributions

XMW contributed to the entire study process from its conception through to data collection, analysis, interpretation, and the preparation of this manuscript. NW and SQL assisted with data collection, analysis, and interpretation; made suggestions regarding the manuscript; and supported the entire study process. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This study was approved by the ethical review board of the second affiliated hospital of Chongqing Medical University (2018 Ethical Review No.16).

Consent for publication

Not applicable.

Competing interests

None.

references

1．Wang AJ, Peng AP, Li BM, Gan N, Pei L, Zheng XL, et al. Natural history of covert hepatic encephalopathy: An observational study of 366 cirrhotic patients. World J Gastroenterol. 2017; 23(34): 6321-9.

2．Bajaj JS, Reddy KR, Tandon P, Wong F, Kamath PS, Garcia-Tsao G, et al. The 3-month readmission rate remains unacceptably high in a large North American cohort of patients with cirrhosis. Hepatology. 2016; 64(1): 200-208.

3．Yildirim M. Falls in Patients With Liver Cirrhosis. Gastroenterol Nurs. 2017; 40(4): 306-310.

4．Vilstrup H, Amodio P, Bajaj J, Cordoba J, Ferenci P, Mullen KD, et al. Hepatic encephalopathy in chronic liver disease: 2014 Practice Guideline by the American Association for the Study of Liver Diseases and the European Association for the Study of the Liver. Hepatology. 2014; 60(2): 715-735.

5．Barone M, Shahini E, Iannone A, Viggiani MT, Corvace V, Principi M, et al. Critical flicker frequency test predicts overt hepatic encephalopathy and survival in patients with liver cirrhosis. Dig Liver Dis. 2018; 50(5): 496-500.

6．Cavaggioni G, Poli E, Ferri F, Parlati L, Monaco V, Melcore C, et al. MELDNa score is associated with psychopathology and reduced quality of life in cirrhotic patients with a liver transplant perspective. Metab Brain Dis. 2017; 32(3): 923-933.

7．Chaney A, Werner K T, Kipple T. Primary Care Management of Hepatic Encephalopathy: A Common Cirrhosis Complication. The Journal for Nurse Practitioners. 2015; 11(3): 300-306.

8．Hanai T, Shiraki M, Watanabe S, Kochi T, Imai K, Suetsugu A, et al. Sarcopenia predicts minimal hepatic encephalopathy in patients with liver cirrhosis. Hepatol Res. 2017; 47(13): 1359-1367.

9．Jepsen P, Watson H, Andersen PK, Vilstrup H. Diabetes as a risk factor for hepatic encephalopathy in cirrhosis patients. Journal of Hepatology. 2015; 63(5): 1133-1138.

10．Tsai CF, Chen MH, Wang YP, Chu CJ, Huang YH, Lin HC, et al. Proton Pump Inhibitors Increase Risk for Hepatic Encephalopathy in Patients With Cirrhosis in A Population Study. Gastroenterology. 2017; 152(1): 134-141.

11．Bajaj JS, Sanyal AJ, Bell D, Gilles H, Heuman DM. Predictors of the recurrence of hepatic encephalopathy in lactulose-treated patients. Aliment Pharmacol Ther. 2010; 31(9): 1012-1017.

12．    Riggio O, Amodio P, Farcomeni A, Merli M, Nardelli S, Pasquale C, et al. A Model for Predicting Development of Overt Hepatic Encephalopathy in Patients With Cirrhosis. Clin Gastroenterol Hepatol. 2015; 13(7): 1346-1352.

13．Tapper EB, Parikh ND, Sengupta N, Mellinger J, Ratz D, Lok AS, et al. A risk score to predict the development of hepatic encephalopathy in a population-based cohort of patients with cirrhosis. Hepatology. 2018; 68(4): 1498-1507.

14．Chen L, Jiang L, Shen A, Wei W. Development of a quality instrument for assessing the spontaneous reports of ADR/ADE using Delphi method in China. Eur J Clin Pharmacol. 2016; 72(9): 1135-1142.

15．    de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005; 27(7): 639-643.

16．Saberifiroozi M. Improving Quality of Care in Patients with Liver Cirrhosis. Middle East J Dig Dis. 2017; 9(4): 189-200.

17．Han S, Zhu R, Cheng J, Yue X. Re-discussing the content of the prescription rights of midwives under certain circumstances using the Delphi method. Chinese Nursing Research. 2017; 4(3): 146-150.

18．Putnam J W, Spiegel A N, Bruininks R H. Future directions in education and inclusion of students with disabilities: a Delphi investigation. Exceptional Children. 1995; 61(n6): 553-576.

19．Albuquerque MR, Lopes MC, de Paula JJ, Faria LO, Pereira ET, da Costa VT. Cross-Cultural Adaptation and Validation of the MPAM-R to Brazilian Portuguese and Proposal of a New Method to Calculate Factor Scores. Front Psychol. 2017; 8: 261.

20．Lakanmaa RL, Suominen T, Perttilä J, Ritmala-Castrén M, Vahlberg T, Leino-Kilpi H. Basic competence in intensive and critical care nursing: development and psychometric testing of a competence scale. J Clin Nurs. 2014; 23(5-6): 799-810.

21．Labenz C, Toenges G, Huber Y, Nagel M, Marquardt JU, Schattenberg JM, et al. Development and Validation of a Prognostic Score to Predict Covert Hepatic Encephalopathy in Patients With Cirrhosis. Am J Gastroenterol. 2019.

22．    Bager P. The assessment and care of patients with hepatic encephalopathy[J]. Br J Nurs. 2017; 26(13):724-729.

23．Tapper EB, Finkelstein D, Mittleman MA, Piatkowski G, Chang M, Lai M. A Quality Improvement Initiative Reduces 30-Day Rate of Readmission for Patients With Cirrhosis. CLINICAL GASTROENTEROLOGY AND HEPATOLOGY. 2016; 14(5): 753-759.

24．Künzler-Heule P, Beckmann S, Mahrer-Imhof R, Semela D, Händler-Schuster D. Being an informal caregiver for a relative with liver cirrhosis and overt hepatic encephalopathy: a phenomenological study. JOURNAL OF CLINICAL NURSING. 2016; 25(17-18): 2559-2568.

25．Gimenes FR, Reis RK, da Silva PC, Silva AE, Atila E. Nursing Assessment Tool for People With Liver Cirrhosis. Gastroenterol Nurs. 2016; 39(4): 264-272.

tables

Table 1. General information of experts in two rounds [n (%)]

Table 2. Consensus of experts in two rounds [n (%)]