Derivation and Validation of Risk Scores for Cause-Specific Mortality using Verbal Autopsy Data

DOI: https://doi.org/10.21203/rs.3.rs-477828/v1

Abstract

Background

Deaths certification remains a challenge mostly in the low-resources countries which results in poor availability and incompleteness of vital statistics. In such sceneries, public health and developmental policies concerning the burden of diseases are limited in their derivation and application. The study aimed at developing and evaluating appropriate cause-specific mortality risk scores using Verbal Autopsy (VA) data. 

Methods

A logistic regression model was used to identify independent predictors of NCDs, AIDS/TB, and CDs specific causes of death. Risk scores were derived using a point scoring system. Receiver operating characteristic (ROC) curves were used to validate the models by matching the number of reported deaths to the number of deaths predicted by the models. 

Results

The models provided accurate prediction results with sensitivities of 86%, 46%, and 40% and false-positive error rates of 44%, 11%, and 12% for NCDs, AIDS/TB, and CDs respectively. 

Conclusion

This study has shown that, in low- and medium-income countries, simple risk scores using information collected using verbal autopsy questionnaire could be adequately used to assign causes of death for Non-Communicable Diseases and AIDS/TB

Background

The need for correct identification of death is very crucial for measuring the burden of diseases in a country. Statistics in mortality and cause of death are an essential component of population health status and are required for identifying priority areas and for policymaking, planning, and programme implementation13. Vital registration (VR) and civil statistics remain the main source of data for demographic and health researchers4, challenges like incomplete and high proportions of ill-defined deaths persists in low- and middle-income countries compared to high-income countries57. This problem is particularly alarming in the rural area environment and less developing countries where it can be very expensive to undertake a proper and accurate assessment. Well-trained personnel and/or doctors also play a crucial role in death certification using the World Health Organization (WHO) standardized international classification of diseases ICD-10 guide. In the populations lacking vital registration and qualified medical certification, Verbal Autopsy (VA) has become a primary source of information on causes of death1.

Verbal autopsy is a procedure that is used in determining the probable cause of death, often in societies where medical certification of death is poor and limited8. It has its origins from the 19th century1,9 and involves the use of detailed questionnaire administrated to collect information from relative or caregivers of the deceased on the symptoms and events during illness leading to death8,10. This is now widely used in the developing world to estimate cause-specific mortality rates, disease surveillance, and sample registration. Data arising from the verbal autopsy procedure has been used to assess time trends, evaluations of health interventions, and to make comparisons in the burden of disease, locally as well as globally. However, the usefulness of the VA data relies on the correct identification and certification of death11.

The use of physician review VA analysis has been validated and has provided useful information to correctly assign the cause of death. An alternate procedure that is outgrowing and been validated is computer-based verbal autopsies analysis, hence physical-certified verbal autopsy is widely used than the Computer-coded verbal autopsy because computer-coded VA is easily implemented by astute individuals. Computer-coded VA analysis ranges from Inter-VA, Tariff to probabilistic method5,9,12−15. To evaluate the effectiveness of both methods, some studies have compared the physician review and computer-coded VA analysis16. Fantahun et.al14 confirmed a strong correlation between the probabilistic methods of assigning the cause-specific and physician review VA analysis. Studies have shown the significance of the probabilistic approach, however, a study by Lozano et al13 showed that the InterVA approach did not perform well in assigning cause-specific compared to Physician-certified VA. To validate probabilistic methods, different studies have been conducted, for example, a study by Vergnano et al, Bauni et al and King et al12,17,18. However, no study had been conducted to develop a risk score for assigning the cause of death using signs and symptoms identified before death. This study aimed at developing and validating appropriate cause-specific mortality risk scores using Verbal Autopsy (VA) data.

Methods

Data

The data for this study are from Population Health Metric Research Consortium (PHMRC) Verbal Autopsy data which is obtainable at http://ghdx.healthdata.org/record/population-health-metrics-research-consortium-gold-standard-verbal-autopsy-data-2005-2011. Data were collected from six sites from four different countries (India, Mexico, Tanzania, and the Philippines) as a part of the Population Health Metrics Research Consortium (PHMRC) project. The data included the demographic characteristics, the period of illness, record of history on a chronic condition, and signs and symptoms. Data was cleaned to make it suitable for our expert algorithm analysis. Murray et al8 provided a full detailed paper on the PHMRC verbal autopsy data.

Cause-specific were divided into three categories, i.e. AIDS/TB, communicable and Non-Communicable Diseases. AIDS and TB were combined to avoid cause misclassification due to TB being the most common opportunistic infection among HIV infected. Communicable diseases included Malaria, diarrhea, Pneumonia, and other Infectious Diseases. Non-Communicable diseases included Diabetes, Cancer, Heart diseases, Stroke, Asthma, Epilepsy, and other unspecified Non-Communicable diseases. 

For analysis, demographic information, signs, and symptoms, and assigned cause-specific were extracted from the dataset. The deceased ages were grouped into 10 to 34 years, 35 to 64 years 65 or older. Since other sites had smaller or even zero death count, sites were classified as Africa sites, Asia sites, and North America. Furthermore, other signs and symptoms were combined to reduce the number of symptoms, for example, fever and sweating were combined; cough and if cough produces sputum were combined. Dar-es-Salaam sample was considered for the analysis since it contained more data points and situated in Africa.

Statistical analysis

Descriptive statistics including frequency count and proportions of deaths were given for each cause specific. Comparisons of proportions were undertaken using the Chi-square test for categorical data. Multivariable logistic regression was used to create a predictive model based on the derivation dataset, which included 1,196, 1,202, and 1,217 NCD’s, AIDS/TB, and CD’s deaths, where all covariates with a p-value less 0.25 were considered for modeling. Validations were done on the 25% datasets of each specific cause using the Africa site data. The discriminating ability of the predicted models was evaluated using the AUROC curves. Sensitivity, specificity, PPV, and NPV were reported for each cut-point and the best cut-point was decided upon. A point scoring system was used to derivate the risk scores associated with a single specific cause. Statistical analyses were undertaken using STATA.

Results

A total of 6224 adult deaths were included in the verbal autopsy dataset across three sites. The majority of deaths were among males accounting for 54.7% while the remaining percentages accounted for female deaths. Of those deceased, 50.9% (3,166) were aged between 35 and 64, and 56.6% (3,213) were from Asia sites. Higher proportions of deaths were attributable to NCD (69%), followed by CD and AIDS/TB deaths. Gender, age at death, and site were statistically significantly associated with deaths Results are presented in Table 1. 

Table 1: Distribution of cause of death by gender, by age and by site.

Characteristics







Total (%)

AIDS/TB (%)

CD's (%)

NCD's (%)

p-value

Overall

6224 (100)

778 (100)

1131 (100)

4315 (100)


Gender






Female

2817 (45.3)

311 (40.0)

49gt5 (43.7)

2011 (46.6)


Male

3398 (54.7)

465 (59.7)

635 (56.2)

2298 (53.2)

0.008

unspecified

9 (0.2)

2 (0.3)

1 (0.1)

6 (0.2)


Age at death






11 to 34

1175 (18.9)

275 (35.4)

347 (30.7)

553 (12.8)


35 to 64

3166 (50.9)

443 (56.9)

461 (40.8)

2262 (52.4)

<0.001

>60

1775 (28.5)

34 (4.4)

309 (27.3)

1432 (33.2)

unspecified

108 (1.7)

26 (3.3)

14 (1.2)

68 (1.6)


Site






Africa sites

1612 (25.9)

312 (40.1)

270 (23.9)

1030 (23.9)


Asia Sites

3213 (51.6)

329 (42.3)

704 (62.3)

2180 (50.5)

<0.001

North America

1399 (22.5)

137 (17.6)

157 (13.9)

1105 (25.6)


 

Factors associated with modeled diseases

Risk factors associated with NCD, AIDS/TB, and CD were assessed using the seventy-five percent data from Dar-es-Salaam. Covariates with a p-value< 0.1 were used for developing the risk scores. The final risk score for NCD modeling included gender, age, cough, blood in the vomit, pale, lump, lost weight and belly protrude. Among NCD deaths, the probability of dying from NCD increased with age. Symptoms like cough and producing sputum (95% CI: 0.34 -1.89), weight loss (95% CI: 0.06 – 0.80), pale (95% CI: 0.07 – 0.86), belly protruding (95% CI: 1.22– 2.24), and developing lump (95% CI: 0.70 – 1.86) increased the risk of NCD deaths while fever without sweating was associated with reduced risk of NCDs deaths. The AUC for the risk scores to correctly predict NCD deaths was 0.81 (95% CI: 0.78 to 0.84). The estimated risk of AIDS/TB deaths decreases with an increase in age. Signs and symptoms that were positively associated with AIDS/TB mortality were fever with or without sweat, cough with or without sputum, and itching skin while belly protrude was associated with the reduced risk of AIDS/TB mortality. The AUC for the risk scores to correctly predict AIDS/TB deaths was also 0.81 (95% CI: 0.78 to 0.85). Risk factors associated with reduced CDs related mortality included gender, age from 10 to 35 years, pale, vomit with blood and stop passing water, while fever with or with no sweat, cough with no sputum, and frequent loss of stool associated with a higher risk of CD deaths. The AUC was 0.74 (95% CI: 0.70 to 0.78), indicating fair predictive ability. The results for risk scores are presented in Table 2. Cross-validation was performed using the developed risk scores and 25% remaining dataset. The predictive ability for three-cause specific deaths performed well with AUC of 0.81 (95% CI: 0.77 to 0.85), corresponding to NCDs, 0.79 (95% CI: 0.73 to 0.85) for AIDS/TB model, and 0.68 (95% CI: 0.61 to 0.75) for CDs related deaths. Results are presented in Table 2.

Developed risk scores

Table 3 presents NCD, AIDS/TB, and CD risk score values defined using the multivariable logistic estimated risk score model. For a lower negative coefficient, the score was -4 while for a higher negative coefficient, the score was a -2. For a lower positive coefficient, the score allocated was a 2 while for a higher positive value, a score allocated was a 4. For each reference category, we allocated the score of 0 and the minimum and the maximum score were calculated for each specific cause. NCD had a minimum score of -8 and a maximum score of 24, AIDS/TB score varied from -4 to 22, and CD score varied from -16 to 18. The discriminating score for the developed risk scores for the three-cause specific decreased compared to the derivation dataset. NCD deaths discriminating score decreased from 0.81 to 0.76 and from 0.81 to 0.75; area under the curve for AIDS/TB and CD deaths decreased from 0.81 to 0.77 and 0.79 to 0.76; and 0.74 to 0.69 and 0.68 to 0.56 for both derivation and validation respectively.

Table 2: Estimated risk scores by three cause-specific mortality 

 

NCD

AIDS/TB

CD

Characteristics

Estimates (95% CI)

Estimates (95% CI)

Estimates (95% CI)

Constant 

-1.27 (-1.60; -0.95)

-2.74 (-3.21; -2.28)  

-1.91 (-2.35; -1.48)

Gender

 

0

 

0

 

0

Male 

Female 

0.23(-0.05; 0.51)

0.240(-0.12; 0.59)

-0.60(-0.98; -0.22)

Age

 

0

 

0

 

0

10 to 35 years

35 to 64 years

1.23(0.91; 1.55)

0.05(-0.34; 0.43)

-0.43(-0.86; -0.01)

≥ 65

2.73(2.22; 3.23)

-1.88(-2.79; -0.99)

-0.57(-1.21; 0.06)

Unspecified

1.22(0.69; 1.76)

0.39(-0.29; 1.07)

0.01(-0.69; 0.71)

Cough and Sputum

 

0

 

0

 

0

No cough, no sputum

Cough and no sputum

-1.13(-1.57; -0.70)

1.34(0.87; 1.82)

0.83(0.31; 1.35)

Cough and sputum

-1.78(-2.21; -1.36)

2.15(1.75; 2.56)

0.21(-0.34; 0.76)

Vomit and Blood

 

0

 

 

0

No vomit, no blood

vomit and no blood

0.20(-0.13; 0.52)


0.25(-0.18; 0.67)

Vomit and blood

1.15(0.38; 1.92)

 

-2.29(-4.32; -0.26)

Itching skin (0:no, 1:yes)

-0.87(-1.50; -0.24)

1.10(0.48; 1.70)


Lost weight (0:no, 1:yes)

0.45(0.09; 0.82)



Pale (0:no, 1:yes)

0.49(0.10; 0.87)


-0.91(-1.37; -0.45)

Lump (0:no, 1:yes)

1.31(0.75; 1.88)



Belly protrude (0:no, 1:yes)

1.80(1.30, 2.29)

-0.80(-1.40; -0.19)

-0.76(-1.64; 0.12)

Fever and Sweating

 

 

0

 

0

No fever, no sweat 

Fever and no sweating


1.01(0.59; 1.44)

0.91(0.43; 1.39)

Fever and sweating

 

0.75(0.33; 1.17)

0.99(0.52; 1.45)

Frequent loose of stool



0.98(0.49; 1.47)

Stop urinating



-0.77(-1.49; -0.05)

Derivation sample

1,196

1,202

1,217

 Area Under Curve (95%CI)

0.81 (0.79; 0.84)

0.81 (0.78; 0.85)

0.74 (0.70; 0.78)

 Validation sample

401

403

408

 Area Under Curve (95%CI)

0.81 (0.77; 0.85)

0.79 (0.73; 0.85)

0.68 (0.61; 0.75)

 

Table 3: Developed risk score values for NCD, AIDS/TB and CD.

 

Specific cause

Characteristics

NCD Score

AIDS/TB Score

CD Score

Gender (Male)

 

 

 

Female

2

2

-2

Age (10 to 35 years)

 

 

 

35 to 64 years

2

2

-2

≥ 65

4

-2

-4

Unspecified

2

4

2

Cough and Sputum (No cough or sputum)

 

 

Cough and no sputum

-2

2

4

Cough and sputum

-4

4

2

Vomit and Blood (No vomit or blood)

 

 

vomit and no blood

2

 

2

Vomit and blood

4

 

-2

Itching skin     (No)

 

 

 

Yes

-2

2

 

Lost weight (No)

 

 

 

yes

2

 

 

Pale (No)

 

 

 

yes

2

 

-2

Lump (No)

 

 

 

yes

2

 

 

Belly protrude (No)

 

 

 

Yes

2

-2

-2

No fever, no sweat

 

0

0

Fever and no sweating

 

4

2

Fever and sweating

 

2

4

Frequent loose of stool (No)

 

 

 

Yes

 

 

2

Stop urinating (No)

 

 

 

Yes

 

 

-2

min

-8

-4

-16

max

24

22

18

Derivation sample

1,196

1,202

1,217

Area Under Curve (95%CI)

0.76 (0.74;0.79)

0.77 (0.74; 0.81)

0.69 (0.65; 0.74)

Validation sample

401

403

408

Area Under Curve (95%CI)

0.75 (0.70;0.80)

0.76 (0.70; 0.83)

0.56 (0.48; 0.64)

Performance characteristics of the risk scoring system: Cut-offs.

The various threshold to predict the probability of correctly assigning cause of death for each risk scoring system were assessed. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for best cut-offs were reported. The best cut-offs were decided based on the high sensitivity and high negative predictive value. With the threshold of greater equal to 6 for correctly assign the NCD cause-specific had a sensitivity of 84%, along with specificity 54%, PPV 34%, and NPV 92% which increases by 2% using the validation set. AIDS/TB and CD deaths threshold had a higher specificity compared to sensitivity. At ≥6 cut value, the probability of correctly assigning AIDS/TB deaths was 44% and a negative predictive value of 86% while a sensitivity of 59% and an NPV of 99% for a ≥4 threshold for associated with CD deaths. The chosen cut-off values for the three studied causes performed better in assigning the correct cause of death on the validation set where NCDs models performed better compared to other cause-specific models. Information is presented in Table 4.

Table 4: Optimum cut-offs for a derivation dataset.

Data type

Cause of death

Cut-off value

Sensitivity (%)

Specificity (%)

PPV (%)

NPV (%)

 

NCD

≥6

84

54

34

92

Derivation set

AIDS/TB

≥6

44

90

55

86

CD

≥4

59

89

7

99

 

NCD

≥6

86

56

36

94

Validation set

AIDS/TB

≥6

46

89

49

88

CD

≥4

40

88

4

99

Discussion

The study has used a binary logistic regression to derivate and validate the risk score of the specific cause of death. The models have been applied to verbal autopsy data collected from Dar-es-Salaam, Tanzania. The site had more data points than the other five sites. Our scoring tool achieved a good predictive ability and can be easily implemented in a setting of identifying the NCD’s death using the VA responses. Cough with or without sputum and itching skin decreases the risk of NCD deaths while age, vomit with blood, weight loss, pale, lump, and protruding of the belly were associated with increased risk of NCD-related mortality. Our risk score for NCD’s deaths performed well (AUC ranging from 0.77 to 0.85). Using a cutoff point of 0.6 and above, a maximum of 86% of a true positive NCD death was correctly assigned while only 56% of the true negative was achieved. A final predicted risk score for AIDS/TB deaths model included gender, age, cough, and cough which produces sputum, itching skin, belly protrude, and fever and fever with sweating. The predictive ability of the AIDS/TB deaths model also performed well (AUC ranging from 0.73 to 0.85). The risk factors for CD included cough with sputum, fever with sweating, only fever, and the frequent loss of stools. The model predictive ability was only sufficient (AUC ranging from 0.60 to 0.75). 

Our tool showed its reliability to assign a probable cause of death. The ability to accurately predict the NCDs, AIDS/TB, and CDs related cause of death had acceptable AUC and true positive error rates were high compared to the accuracy of Inter-VA to assign adults deaths13,17. The true positive rates for our scoring system are comparable to Flaxman et al19 with higher specificity observed. According to Desai et al20, the positive predictive value for computer-coded VA, and physician-coded VA ranged from 44 to 69, which are higher compared to our PPV for all three models. Additionally, sensitivity and PPV for accurately assign AIDS/TB related deaths was lower compared to sensitivity and PPV reported by Polprasert et al21 however it is comparable to Lopmari et al22 Though some methods outperformed our method to assign a cause of death, the strength is that this method can be used to assign an individual cause of death23-24. this study was subjected to some limitations. One of the limitations is that accurate information of signs and symptoms depends on the ability of the relative or caregiver to recall the information before one’s death. Most African and South-East Asian countries lack complete death registration data and even worse do not record cause-of-death data. In most cases, Verbal Autopsy is the only option to assess causes of deaths. We have shown that simple risk scores using information collected in the verbal autopsy questionnaire could adequately be used to assign causes of death, especially in developing countries. 

Declarations

Acknowledgement

We would like to acknowledge the PHMRC for making the data assessable for everyone.

Authors contributions

TMN researched, extracted the data, performed data management, statistical analysis and wrote the initial draft of the manuscript. SOMM conceptualised the study and the statistical analysis, helped with the methodology, and supervised the statistical analysis.

Competing interests

All authors declare that they have no competing interests

References

1.        WHO_CDS_CSR_ISR_99.4.

2.        Kamali A, Wagner H, Nakiyingi J, Sabiiti I, Kengeya-kayondo JF, MULDER Kamali A DW. Verbal Autopsy as a Tool for Diagnosing HIV-Related Adult Deaths in Rural Uganda. Vol 25.; 1996. http://ije.oxfordjournals.org/

3.        Soleman N, Chandramohan D, Shibuya K. Verbal Autopsy: Current Practices and Challenges. Vol 84.; 2006.

4.         Garenne M. Prospects for Automated Diagnosis of Verbal Autopsies.

5.        Byass P, Kahn K, Fottrell E, Mee P, Collinson MA, Tollman SM. Using verbal autopsy to track epidemic dynamics: The case of HIV-related mortality in South Africa. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-46

6.        United Nations. World Mortality Report 2015 - Highlights.; 2015.

7.       Joint United Nations Programme on HIV/AIDS. 2007 UNAIDS Annual Report : Knowing Your Epidemic. World Health Organization; 2008.

8.        Murray CJL, Lopez AD, Black R, et al. Population Health Metrics Research Consortium gold standard verbal autopsy validation study: Design, implementation, and development of analysis datasets. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-27

9.       Murray CJL, Lozano R, Flaxman AD, Vahdatpour A, Lopez AD. Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-28

10.     Setel PW. Verbal autopsy and global mortality statistics: If not now, then when? Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-20

11.      Baiden F, Bawah A, Biai S, et al. Setting international standards for verbal autopsy. Bulletin of the World Health Organization. 2007;85(8):570-571. doi:10.2471/BLT.07.043745

12.    King G, Lu Y, Shibuya K. Designing Verbal Autopsy Studies.; 2010. http://gking.harvard.

13.      Lozano R, Freeman MK, James SL, et al. Performance of InterVA for assigning causes of death to verbal autopsies: Multisite validation study using clinical diagnostic gold standards. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-50

14.      Fantahun M. MORTALITY AND SURVIVAL FROM CHILDHOOD TO OLD AGE IN RURAL ETHIOPIA.

15.      Reeves BC, Quigley M, C RB. A Review of Data-Derived Methods for Assigning Causes of Death from Verbal Autopsy Data. Vol 26.; 1997.

16.      Murray CJL, James SL, Birnbaum JK, Freeman MK, Lozano R, Lopez AD. Simplified Symptom Pattern Method for verbal autopsy analysis: Multisite validation study using clinical diagnostic gold standards. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-30

17.      Bauni E, Ndila C, Mochamah G, et al. Validating physician-certified verbal autopsy and probabilistic modeling (InterVA) approaches to verbal autopsy interpretation using hospital causes of adult deaths. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-49

18.     Vergnano S, Fottrell E, Osrin D, et al. Adaptation of a probabilistic method (InterVA) of verbal autopsy to improve the interpretation of cause of stillbirth and neonatal death in Malawi, Nepal, and Zimbabwe. Population Health Metrics. 2011;9. doi:10.1186/1478-7954-9-48

19.      Flaxman AD, Joseph JC, Murray CJL, Riley ID, Lopez AD. Performance of InSilicoVA for assigning causes of death to verbal autopsies: Multisite validation study using clinical diagnostic gold standards. BMC Medicine. 2018;16(1). doi:10.1186/s12916-018-1039-1

20.       Desai N, Aleksandrowicz L, Miasnikof P, et al. Performance of Four Computer-Coded Verbal Autopsy Methods for Cause of Death Assignment Compared with Physician Coding on 24,000 Deaths in Low-and Middle-Income Countries. http://www.biomedcentral.com/1741-7015/12/20

21.      Polprasert W, Rao C, Adair T, Pattaraarchachai J, Porapakkham Y, Lopez AD. Cause-of-death ascertainment for deaths that occur outside hospitals in Thailand: Application of verbal autopsy methods. Population Health Metrics. 2010;8(1). doi:10.1186/1478-7954-8-13

22.     Lopmari B, Cook A, Smith J, et al. Verbal autopsy can consistently measure AIDS mortality: A validation study in Tanzania and Zimbabwe. Journal of Epidemiology and Community Health. 2010;64(4):330-334. doi:10.1136/jech.2008.081554

23.      Mccormick TH, Li ZR, Calvert C, et al. Probabilistic Cause-of-Death Assignment Using Verbal Autopsies *.; 2014.

24.     Serina P, Riley I, Stewart A, et al. Improving performance of the Tariff Method for assigning causes of death to verbal autopsies. BMC Medicine. 2015;13(1). doi:10.1186/s12916-015-0527-9