In-depth Validation of CHERG’S Verbal Autopsy-Social Autopsy (VASA) Tool for Ascertaining Determinants and Causes of Under-Five Child Mortalities in Karachi, Pakistan.

doi:10.21203/rs.3.rs-35489/v1

Download PDF

Research article

In-depth Validation of CHERG’S Verbal Autopsy-Social Autopsy (VASA) Tool for Ascertaining Determinants and Causes of Under-Five Child Mortalities in Karachi, Pakistan.

https://doi.org/10.21203/rs.3.rs-35489/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Globally, child mortality estimates are more clustered among the developing countries where quality data on estimates and determinants of child mortality are compromised. To achieve sustainability in reducing child mortality estimates, the integrated Verbal Autopsy and Social Autopsy (VASA) tool help in estimating prevalence and assigning medical and social causes and determinants of child survival, especially in the developing countries. A validation study of the Child Health Epidemiology Reference Group’s (CHERG) Verbal autopsy/Social Autopsy (VASA) tool has been undertaken for employing in a Karachi VASA Integrated Child Mortality Investigation-ICMI study in its urban slums.

Methods: Validity and reliability of the CHERG VASA-tool were tested using face, content, discriminant validation and reliability tests on one hundred randomly selected mothers, with a recent child death event. Data were computed on SPSS (version-21) and R.

Results: Testing yielded high I-CVI (>81.43%); high Cronbach's Alpha (0.843); accuracy of between 75% and 100% of the discriminants classifying births to live and stillbirths. The tool showed ICVI (>82.07% and 88.98% respectively) with high accuracy (92% and 97% respectively) for assigning biological and social causes of child deaths respectively.

Conclusion: The CHERG VASA questionnaire is valid, reliable, and relevant to the conceptual framework. This valid tool is one of the assets for child health policy as it can assign accurate medical and non-medical causes (pertaining to health-seeking practices) of child mortality cases occurring in Pakistan.

Pediatrics

Children

under-five

mortality

Verbal Autopsy

Social Autopsy

Validation

VASA

Verbal autopsy (VA) and Social autopsy (SA) are the techniques that are used to assign medical causes and social reasonings, respectively, to the child death events in low resource settings(1–3). These techniques not only provide extended information but indirectly, also assists in updating the national Civil Registration and Vital Statistics (CRVS) and national Health Management Information System (HMIS) on the mortalities which have been missed to be recorded by CRVS (4, 5). Moreover, VASA, an integrated approach, whereby VA is merged with SA to identify not only the medical causes of mortalities but also to identify the social contributors of child deaths. This integrated tool is under trial in some of the low resource settings across the globe(4, 6–9). Literature suggests that CHERG (Child Health Epidemiology Reference Group) VASA integrated tool, which is based on the so-far most holistic conceptual framework, i.e., “The Pathway to Survival Conceptual Framework” (TPtoSCF)(10), have been utilized in several countries(6, 7, 9, 11) for exploring the most relevant medical and social determinants of mortality events. This information ultimately helps the epidemiologists in assigning the medical and social causes to child deaths. The extensive data obtained from VASA tool based investigations provides highly elaborated evidence and information on extended determinants of child mortality which is of great public health importance(4, 6, 12).

Similar to any data collection tool, it’s always been a concern of whether the data collected through VA and SA are correct, reliable, and represents information what the researcher intends to collect. The tool with exceptionally high validity and reliability can only help to capture the correct data which is of public health importance. The process of validation involves testing the questionnaire on these parameters. In order to obtain best possible data, many efforts have been undertaken globally in refining the methodology of VA(13–19). In continuation of such efforts, a few of the validation studies have been undertaken globally as well as in Pakistan to assess the capacity of VA tool in providing precise and reliable evidence on the causes of child mortality(13, 14, 18, 19). However, no attempts have been made in validating the SA, as well as the integrated VASA tool. Keeping in view of the current status of child mortality estimates in Pakistan (which stresses upon the fact that the country is among the top ten countries which hold significant burden of child mortality across the globe(20–22)), a VASA ICMI study has been undertaken in Karachi. The primary aim of VASA ICMI study was to gather evidence on a comprehensive list of determinants related to under-five mortality in the underserved locations where there is a high chance that the child mortality cases might be missed to be registered under national CRVS and HMIS. However, prior to data collection, the CHERG’s VASA tool (which was chosen to be used in the ICMI) was recently validated. This validation study is one of its kind, as no VASA integrated tool has been validated so far in such detail by focusing most of the techniques for measuring validity and reliability of the tool. Also, since the CHERG's VASA integrated tool is based on a most holistic conceptual framework, i.e., "The Pathway to Survival Conceptual Framework" (TPtoSCF), we hope that the validated tool will be highly beneficial in providing extended and detailed data on the most relevant determinants and causes of under-five mortality in Karachi, Pakistan. In this paper, we share the validation results of the CHERG VASA tool.

This validation study is a part of Karachi’s VASA ICMI study(23) and objectively focuses assessing the validity and reliability estimates of the CHERG VASA questionnaire, which was later utilized in Karachi’s VASA ICMI study.

The sample size

Literature shows very succinct specific guidelines(24, 25) on the sample size required for pre-survey validation exercise of VASA tool, however, general guidelines suggests(25) that generally the sample size needed for any questionnaire validation should be a minimum of 20% of the study’s total sample size. The sample size of the Karachi VASA ICMI study was 400 under-five deceased cases; therefore 20% of 400 cases, i.e., 80 cases (extra to 400 cases of the ICMI study) were recruited from the community for the questionnaire validation study. This figure was later inflated to 100 cases. Mothers of these 100 under-five deceased cases (n = 100) were randomly selected from 12 selected slums across Karachi.

Inclusion and exclusion criteria

Only those mothers were included in this validation process, whose under-five children died within past six weeks (to minimize response bias) due to any illness and those who have death certificate (issued by health authority or health care practitioner/s) with the cause of death (CoD) mentioned on the death certificate. The CoD mentioned on the death certificate helped us in verifying the level of accuracy of the assigned medical-CoD (MCoD) by the physician (based on data from VA component of the VASA tool using a prior algorithm(26) as well as based on their own expertise).

Ethical clearance for VASA ICMI study

The ethical approval of this study was gained from the Medical Ethics Committee, University Malaya Medical Center (201412-847) and Advance Educational Institute and Research Centre (MU/ECA/48/776).

Data collection for the validation study

Trained staff visited the included households and filled the questionnaire during direct interviews. Collected data were shared with verbal autopsy reviewers for Physicians certified VA technique (PCVA) and Algorithm certified VA technique (ACVA). Data and CoD assigned by physicians were computed on SPSS (version 21) and R. The CoD assigned by physicians were compared with CoD mentioned on the death certificates which served as the gold standard diagnosis of the CoD.

Structure of the CHERG VASA Questionnaire

The CHERG VASA questionnaire is a tool for collecting the data on the history of events right from the conception of the mother until the death of the child. It also captures the signs and symptoms a child encountered during the illness. The initial pre-survey CHERG VASA questionnaire consisted of VA and SA components having 14 different modules focusing the different variables ranging from demographic profile and socioeconomic status to variables encompassing the entire continuum of care during maternal pregnancy till the death of the child.

After the section on the VASA general information, the VA component has 06 modules altogether, while the SA component has 07 modules (table-1, column-1&2). These components (VA and SA) and modules have been arranged in such an order that as the interview proceeds, the sequence of events gets unfold (table-1, column-3). To record relevant information of relevant age group (i.e., neonatal deaths, and post-neonatal deaths), specific skip patterns exists in the tool.

Table-1: Structure of CHERG VASA questionnaire

Details of different modules of VA & SA components included in VASA tool

Sequence of different modules of VA & SA components in VASA tool

VASA General Information:

● Deceased background,

● Interview details,

● Consent,

● Respondent’s details,

● Details of others present during interview

VA Sections:

VA-1(background)

VA-2 (maternal History)

VA-3 (neonatal deaths)

VA-4 (infants and child deaths)

VA-5 (health records)

VA-6 (open ended questions)

SA Sections:

SA-1(mother and her household)

SA-2 (social capital)

SA-3 (maternal history)

SA-4 (care-seeking for maternal complications)

SA-5a (care of newborn)

SA-5b (preventive care of post-neonates)

SA-6 (care-seeking of child’s fatal illness)

SA-7 (open ended questions)

(VA-1); (SA-3); (VA-2); (SA-4); (SA-5a); (VA-3); (SA-5b); (VA-4); (SA-6); (VA-5); (SA-1); (SA-2); (VA-6) & (SA-7).

Translation & Back Translation:

Before validation, the questionnaire was forward translated into the local language, Urdu, and then back-translated to English based on “Brislin Back-translation” technique(27). The process involved a professional language translator and a panel of professional bilingual subject experts (paediatrician, gynaecologist, obstetrician, social scientist, and psychologist). The final translated version was processed for the validation process.

Tool’s validation procedure:

As illustrated in table-2 and diagram-1, the CHERG VASA tool was evaluated on its capacity to measure and produce true and actual results, what it intends to measure-this is called the validity of the tool; and on its ability to give the same results every time it will be run elsewhere, this is called as reliability of the instrument. The complete process of this tool’s validation exercise included face validation, content validation, discriminant validation, and reliability analysis is described in respective sections below.

Table-2: Summary of methodology for validation of CHERG VASA questionnaire
Assessment	Method	Outcome
Face Validation	Subjective review of the tool by a panel of experts	The updated questionnaire which was assessed on C.V., D.V., & R.A. (below in this table)
Content Validation (C.V.)	Calculation of I-CVI using agreement level of 05 experts on a dichotomous scale	I-CVI ranged between 81.43–96.36%.
Discriminant Validation (D.V.)	Calculation using Eigenvalue (E.V.), Canonical correlation (C.C.), Wilk’s Lambda (W.L.) and Accuracy	● E.V.=between 0.61 to 735.11 ● C.C.= between 0.86 and 1.00 ● W.L.=<0.001 ● Accuracy of > 75.07%
Reliability Analysis (R.A.)	Calculation using Cronbach’s Alpha (C.A.) statistics	C.A ranged between 0.843 and 0.973
C.V.: Content validation; D.V.: Discriminant Validation; R.A.: Reliability Analysis; I-CVI: Item-content Validity Index; E.V.: Eigenvalue; C.C.: Canonical correlation; W.L.: Wilk’s Lambda; C.A.: Cronbach’s Alpha;

Face Validation. Here, the questionnaire was reviewed and critically evaluated by the same panel (utilized during the translation exercise) on the tool’s format to find out, whether all of the sections and items be easily readable, clear and straightforward to understand and comprehend by the interviewee and respondent; and on the need of rephrasing, omitting or adding any questions. The item was phrased in such a way that the respondent would need to think before responding, thereby minimising the chances of response bias. The panellists found no need for any modifications in terms of rephrasing of questions, omitting any option of any item, adding any option of any item, etc. The translated questionnaire which went for face validation was then named as “VASA Questionnaire-version-1”. This version of the questionnaire was utilised for the content validation as well as to collect data for undertaking discriminant validation.

Content validation involved five subjective reviews, each with five homogenous experts from each professional capacity (five paediatricians, five gynaecologists, five obstetricians, five social scientists, and five psychologists). Each member from a panel was requested to rate every item on the VASA Questionnaire-version-1 on a 5-point likert scale based on the relevancy to the conceptual framework (1 = not relevant, 2 = somewhat relevant, 3 = relevant, 4 = quiet relevant, 5 = highly relevant). The ratings were calculated using Item-content Validity Index (I-CVI). For each item, the I-CVI was computed as the number of experts rated an item on top two boxes (i.e. quiet relevant or highly relevant), divided by the total number of experts. The level of agreement between experts should be as high as possible to make the I-CVI acceptable. The data on content validation is presented in table-3 under different themes/constructs. These themes/constructs were clustered either as pre-birth themes/constructs and post-birth themes/constructs, Sect. 1 through 8 covers pre-birth constructs and Sect. 9 through 16 covers during and post-birth constructs. The average rating of each section was more than 4 (on a scale of 1 to 5), and the I-CVI is more than 80% for all the sections of the VASA questionnaire. Relatively lower I-CVI was observed for “symptoms before pregnancy” (81.43%) and “Birth History of Child” (81.82%) sections.

Table-3: Average expert rating and overall I-CVI of themes of VASA questionnaire
VASA questionnaire themes	Content Validity (C.V.) (average rating)					Average I-CVI
VASA questionnaire themes	Rater 1 (n = 5)	Rater 2 (n = 5)	Rater 3 (n = 5)	Rater 4 (n = 5)	Rater 5 (n = 5)	Average I-CVI
*Content validation (C.V.) of pre-birth themes/constructs of VASA questionnaire**
Demographic and social constructs	4.56	4.44	4.47	4.53	4.53	92.94%
Symptoms before pregnancy	4.43	4.21	4.14	4.21	4.29	81.43%
Symptoms during initial two trimesters	4.45	4.24	4.27	4.24	4.36	82.42%
Symptoms during last trimesters	4.38	4.38	4.38	4.38	4.38	87.50%
Care Seeking behavior for symptoms of last trimester	4.64	4.45	4.50	4.45	4.50	96.36%
Symptoms during labor and care-seeking behavior	4.50	4.30	4.40	4.40	4.50	88.00%
Treatment of labor symptoms and post-delivery	4.34	4.16	4.19	4.22	4.28	83.75%
Knowledge Attitude and Behavior on maternal symptoms	4.53	4.39	4.47	4.43	4.49	88.98%
*Content validation (C.V.) of during & post-birth themes/constructs of VASA questionnaire**
Birth History of Child	4.41	4.27	4.27	4.23	4.36	81.82%
Physical Malformations Of Child's Body	4.57	4.43	4.36	4.43	4.50	88.57%
Child Illness Details	4.31	4.17	4.14	4.17	4.21	82.07%
Health Seeking Behavior Of Child Illness	4.63	4.50	4.56	4.56	4.56	95.00%
Health care professionals (HCP) involved during Child Illness	4.46	4.21	4.36	4.32	4.36	86.43%
Services Cost Burden and Delays	4.50	4.21	4.29	4.29	4.43	91.43%
Preventive Measures Between Child Birth and Illness	4.68	4.55	4.64	4.55	4.68	92.73%
Community Efforts during child illness	4.42	4.21	4.26	4.26	4.32	85.26%
Cause of death-Medical	4.31	4.17	4.14	4.17	4.21	82.07%
Cause of death-Social	4.53	4.39	4.47	4.43	4.49	88.98%
Based on a thorough review of the questionnaire, content of the VASA questionnaire can be divided into two clusters, Sect. 1 through 8 covers pre-birth constructs and Sect. 9 through 16 covers during and post-birth constructs. C.V.: Content validation; I-CVI*: Item-content Validity Index;

Discriminant validation (construct validity). It shows the distinctiveness of different constructs within the questionnaire and helps to identify whether the items on the questionnaire that are not supposed to be related to each other are actually unrelated or not. It is one of the types of construct validity. The “VASA Questionnaire-version-1” (after the face validation exercise) of the tool was utilised for data collection. Collected data were shared with verbal autopsy reviewers (physicians). The reviewers used their clinical reviewing expertise (process called “Physicians certified VA technique-PCVA”) and also used an algorithm developed in prior (process called “Algorithm certified VA technique-ACVA”) in assigning the cause of deaths. The complete data (including the CoD assigned by the physicians) were computed on SPSS (version 21) and R for the validation process. For pre-birth constructs, “birth status of the child” and for during and post-birth constructs “diseases or conditions directly leading to death” are the decisive elements with potential discriminating power. Clear clusters and grouping in response pattern on VASA elements can be seen in diagram-2 for pre-birth constructs. Therefore, valid response differences between the live birth group and stillbirth group (on the elements of each section) were used for the discriminant validity of the pre-birth constructs of the tool. Similarly, valid response differences among different diseases or conditions directly leading to the death were used for the discriminant validity of during or post-birth constructs of the tool.

For pre-birth constructs of the questionnaire, discriminant analysis was performed using birth status as a dependent variable, which was defined in such a way that birth status was coded as “1” (one) in case of stillbirth and “0” (zero) in case of live birth, and elements of each section as discriminating or classification variables. The goodness of fit summary of discriminant analysis models, such as eigenvalue (E.V.) of canonical discriminant function, canonical correlation (C.C.), and Wilks’ Lambda (W.L.) statistics were used for the discriminant validity measures of the constructs. To analyze the discriminant validity of post-birth constructs of VASA questionnaire, multinomial logistic regression was performed using “disease or condition directly leading to death” as dependent variables and elements of each section of the questionnaire as explanatory variables. Likelihood-Ratio (LR) test and Pseudo R-Square measures (Cox and Snell, Nagelkerke, and McFadden) were used as discriminant validity measures of the constructs.

Out of 100 cases, we had 33 stillbirths and 67 live birth cases. For the analysis, unequal prior probabilities of stillbirth and live birth were assumed and assigned prior probabilities as per the group size in data. Backward variable selection method was used to compute the discriminant analysis model. It is vital to comment here at this point that the variables which are not included in the final model do not necessarily mean they are not important, but it means these variable(s) does not provide any additional statistical evidence towards the classification of live and stillbirth.

The goodness of fit summary of discriminant analysis models of each of the pre-birth constructs of VASA questionnaire is presented in table-4. The eigenvalue (E.V.) of canonical discriminant function is the ratio of between groups sum of squares to within groups sum of squares. A high value of eigenvalue indicates that the canonical function explains sufficiently higher variation in data. Similarly, canonical correlation (C.C.) coefficient shows the predictive ability of canonical discriminant function. The value of C.C. ranges from 0 to 1, with a value closer to 1 shows the strong predictive ability of canonical discriminant function. Third goodness of fit statistics i.e. Wilks’ Lambda (ranging from 0 to 1), was calculated to be the proportion of the total variance in the discriminant scores not explained by differences among the groups. A small value of W.L. and statistically significant (p-value of < 0.05) shows the good fit of the discriminant model.

The goodness of fit summary of multinomial logistic regression models of each of during & post-birth constructs of VASA questionnaire is presented in table-4. Statistically significant Likelihood-Ratio (LR) test result infers statistically significant improvement over the null model. Pseudo R-Square measures (Cox and Snell, Nagelkerke, and McFadden) ranging from 0 to 1, with a value closer to “1” (one) indicate good model fit.

Table-4: Discriminant validation & Reliability Analysis of VASA questionnaire
The Goodness of fit summary of discriminant analysis models							Reliability Analysis
VASA questionnaire section	E.V.	C.C.	Accuracy	Wilks' Lambda			C.A.
VASA questionnaire section	E.V.	C.C.	Accuracy	W.L.	Chi-square	p-value	C.A.
Demographic and social constructs	37.44	0.987	100%	0.026	339.37	< 0.001	0.843
Symptoms before pregnancy	2.76	0.86	91.00%	0.266	127.74	< 0.001	0.909
Symptoms during initial two trimesters	2.76	0.86	91.00%	0.266	127.74	< 0.001	0.944
Symptoms during last trimesters	3.146	0.871	92%	0.241	422.91	< 0.001	0.894
Care Seeking behavior for symptoms of last trimester	0.61	0.615	85%	0.621	278.97	< 0.001	0.876
Symptoms during labor and care seeking behavior	5.486	0.92	100%	0.154	446.68	< 0.001	0.954
Treatment of labor symptoms and post delivery	735.11	0.993	100%	0.013	646.35	< 0.001	0.933
Knowledge Attitude and Behavior on maternal symptoms	9.748	0.952	100%	0.093	682.34	< 0.001	0.940
The Goodness of fit summary of multinomial logistic regression models
During and post-birth constructs	LR Tests (p-value)		Accuracy	Cox and Snell	Nagelkerke	McFadden
Birth of Child	< 0.001		100%	0.968	0.996	0.963	0.907
Physical Malformations Of Child's Body	< 0.001		75%	0.968	0.996	0.963	0.947
Child Illness Details	< 0.001		100%	0.968	0.996	0.963	0.948
Health Seeking Behavior Of Child Illness	< 0.001		100%	0.968	0.996	0.963	0.945
Health care professionals (HCP) involved during Child Illness	< 0.001		100%	0.968	0.996	0.963	0.928
Services Cost Burden and Delays	< 0.001		100%	0.956	0.983	0.87	0.95
Preventive Measures between childbirth and illness	< 0.001		100%	0.968	0.996	0.963	0.973
Community Efforts during child illness	< 0.001		100%	0.956	0.983	0.87	0.942
Cause of death-Medical	< 0.001		92%	0.956	0.963	0.963	0.953
Cause of death-Social	< 0.001		97.1%	0.964	0.985	0.89	0.973
C.V.: Content validation; D.V.: Discriminant Validation; R.A.: Reliability Analysis; I-CVI: Item-content Validity Index; E.V.: Eigenvalue; C.C.: Cannical correlation; W.L.: Wilk’s Lambda; C.A.: Cronbach’s Alpha;

A high eigenvalue of canonical discriminant function (> 1), a high value of canonical correlation (> 0.8) and a statistically significant Wilks’ Lambda, with a p-value of < 0.001, validates the good fit of discriminant models for pre-birth constructs of VASA questionnaire. A high value of Pseudo R-Square measures (> 0.8) and a statistically significant LR test, with a p-value < 0.001, validates the good fit of multinomial logistic regression models for during & post-birth constructs of the VASA questionnaire. Hence, underlying constructs of the VASA questionnaire, in all sections (table-4), are distinctive and well-established to address the conceptual framework. More than 85.07% accuracy of the discriminant functions and the multinomial logistic regression model was observed in all sections of the VASA questionnaire.

Reliability analysis: Reliability analysis measure the tool’s capacity to provide the consistent result. The use of Cronbach's Alpha statistics was utilised to measure the reliability of constructs. Due to dominantly dichotomous nature of data, calculation of Cronbach's Alpha based on Pearson correlations matrix would be incorrect or potentially very biased. Therefore, “hetcor” function of “polycor” package of R was used to compute the appropriate heterogeneous correlation matrix. The ‘hetcor’ function is capable of calculating Pearson correlations (for numeric data), polyserial correlations (for numeric and ordinal data), and polychoric correlations (for ordered or non-ordered factors). Reliability for the different sections of the VASA questionnaire is presented in table-4. The universal rule of thumb is that, if the value of Cronbach's alpha (C.A.) is of 0.70 and more, the reliability is good, if the value of C.A. is between 0.80 and 0.90 the reliability is better, and if the value is 0.90 and beyond, the reliability is best. In our study, the Cronbach's Alpha values are above 0.90 across all the constructs of pre-birth, during & post-birth constructs of VASA questionnaire (except “Demographic and social constructs” and “Care Seeking behaviour for symptoms of last trimester” which are 0.843 and 0.876 respectively)(table-4).

Validation studies usually encounter daunting challenges. Most of them include recall biases, information biases, small sample size estimation, etc. To overcome such challenges, the methodology involved in the validation studies should be very concrete and categoric, ensuring that the responses from the validated tool should be able to give information with acceptable accuracy as per the reality of the situation. It is highly essential to undertake such relevant validation studies in developing countries so that it will make the validated version of the VASA tool available for conducting VASA investigations.

Validation of the VA tool

Literature shows several attempts of VA validation studies which have utilised different methodologies to overcome such challenges and identified the precise CoD with much accuracy. The most commonly used VA tool which underwent validation is the World Health Organization’s (WHO) VA tool.

Sample size: Having precisely calculated and appropriate sample size, the results of any validation study show true validity and reliability of the tool. Compared to our validation study, which included 100 deceased under-five cases, all the other VA validation studies underto validation on comparatively larger sample size. We feel that having a smaller sample size, is one of the limitations of our validation study. However, our sample size is purely based on general recommendations from Hertzog(25), which suggests that the minimum sample size for any validation study should be 20% of the sample size calculated for the survey. Aggarwal et al. (13) included 313 neonatal deaths; Nausheen et al.(18) undertook validation on 204 neonatal deaths; the sample size in Soofi et al.(19) was 626 neonatal deaths, and Marsha et al.(14) used 137 neonatal deaths.

Recalling past events: Conducting interviews after a long delay subsequent to the death event may affect the quality and reliability of data. This is most likely because the respondent needs to recall the events and happenings of different events that occurred before and around the death even. It is highly recommended that VA interviews should be undertaken somewhere between 01 months till 06 months provided the family have come out of the mourning period(28). Conducting interviews within the mourning period may cause distress and influence the respondent’s disposition and ability to engage in VA interview(28). P Serina et al.(28), suggests that a long recall period may limit the respondent’s ability in recalling and recollecting pertinent facts. Events (or symptoms) with extraordinary severity (or implications) remain in the recall for a much more extended period compared to those with mild to moderate severity(28). Likewise all the other studies(14, 18, 19, 29), our study undertook interviews within the six (06) weeks after the mortality event. In our sample size, none of the respondents had any difficulty in recalling the events.

Tool’s accuracy: The capacity of any questionnaire to correctly assign the proper CoD to any mortality (due to any specific reason) with a degree of accuracy is called accuracy. When dealing with individual-level data, the diagnostic accuracy of VA is considered to be satisfactory when the specificity and sensitivity of the questionnaire are a minimum of 90%(30). However, at a much larger, i.e. national level, if the validation attempt of VA shows sensitivity and specificity of at least 50% and 90% respectively, the tool is usually considered to be having acceptable diagnostic accuracy(30). Our validation study defines specificity as the ability of the VA section of CHERG’s VASA questionnaire to accurately and correctly assign the medical cause of death compared to the assigned CoD on the medical certificate as well as the based on the signs and symptoms. The closeness of Medical CoD between the one assigned in the death certificate and the one assigned through VASA inquiry was used to measure the validity of the tool. Our study showed accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively. In comparison, the accuracy of the WHO’s VA tool validated in Aggarwals’ study(13) in identifying different neonatal causes of mortalities was found to up to between 78–92% (except for birth asphyxia-16%); however, the kappa statistics was moderate from 0.46 to 0.55. Similarly, in Soofi et al. (19) the WHO VA tool showed the sensitivity of more than 83.5% for diagnosing different causes of neonatal mortality (except congenital malformation, which was 57%), while the specificity of all the major causes of neonatal deaths was found to be more than 90%. Marsha et al.(14) found the sensitivity of the process of assigning the cause of death ranging from 39% (in diagnosing infection) to 90% (in diagnosing causes related to too early/too small syndrome) and specificity ranging from 67% (in diagnosing infection) and 99% (in diagnosing Neonatal Tetanus).

Validation of SA and VASA tools

The literature shows a lack of evidence on validation studies for SA and integrated VASA tool for under-five years children. More specifically, global as well as local (Pakistan specific) literature is lacking on any attempt to endorse the level of validity and and reliability of CHERG VASA questionnaire in diagnosing mortalities of under-five children (born as still and live births) and reported as deceased in a community-based household survey with events reported as full illness-related history. Although the CHERG VASA tool has been designed by experts, still its validation for developing country like Pakistan (where the huge estimates of under-five mortality lies) is technically required, ensuring the availability of a VASA tool that may give the valid and reliable data pertaining to child mortality determinants specifically operating in Pakistan. Such tool is highly required to overcome the data shortage related issues in the country, where the birth and death registration are highly compromised(3, 31, 32). The existing data is also very limited and with questionable validity and accuracy(3, 31, 32). Utilization of this validated tool in identifying mortality related determinants and in assigning medical and social causes of child deaths, especially of those deaths which have been missed from the National Civil Registration and Vital Statistics will surely help us in overcoming data related issues pertaining to child mortalities.

Strengths of our validation study

Our study is first in its kind as it validates the CHERG’s integrated VASA tool, which is based on the most holistic conceptual framework, i.e. TPtoSCF (that addresses the barriers and limitations involved in accessing health care services), thereby recording information required to develop policy in preventing under-five death incidents and improving the child survival estimates in developing countries (like Pakistan) where an unknown, but large number of child mortality events have known to be missed and their medical causes have not been assigned.

Moreover, this study is highly important as it attempted a detailed validation of CHERG’s integrated VASA tool by incorporating rigorous methodology and utilising almost all the core methods suggested for identifying the validity and reliability evidence of any questionnaire. The validation exercise yielded high I-CVI (> 81.43%), with Cronbach's Alpha reliability statistics is 0.843 for “Demographic and social constructs” and highest of 0.973 for “Preventive Measures between childbirth and illness”; accuracy of between 81% (for “Care seeking behavior for symptoms of last trimester”) and 100% of the discriminants classifying births to live and stillbirths. The tool showed accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively.

Although different VA questionnaires have been validated in Pakistan, however, some of them are specifically designed for the neonatal population(19), while others are for stillbirths(18). The CHERG’s VASA integrated questionnaire encompasses three age groups and has not been validated as for the Pakistani population.

An electronic CAPI (Computer Assisted Personal Interview) notebook format for data collection with skip patterns using CSPro software application has also been developed for this validated tool which was utilized for data collection during the VASA ICMI survey. The primary purpose for transforming the lay paper-based form to notebook application is to minimize the data entry errors.

Limitations of the study

Although we undertook our validation study on the recommendations that sample size for validation should be at least 20% of the sample size chosen for survey, still our sample size is smaller.

Recommendations

Future research should be geared to undertake the validation of different VA and SA tools (as separate tools). This should be coupled with the efforts on enhanced research on the efficacy of SA tool and to improve its capacity and accuracy for identifying social determinants pertaining to under-five mortality. Moreover, there is a strong need to have specific recommendations for estimated sample size explicitly required for the validation process of the VASA tool. This would help future researches to undertake analyses based on standardized recommendations. Additionally, it would be much better to conduct quantitative and qualitative assessment on the respondents point of view and their feedback towards the tool.

Based on our results, we confirm that the CHERG VASA integrated questionnaire is valid, reliable, and relevant to the conceptual frameworks. The modified CHERG VASA tool can be utilized for establishing medical and social causes of Under-five child deaths in Karachi, Pakistan and is one of the assets for Pakistan’s child health policy with the potential to record relevant data and assign causes of child mortality cases occurring in Pakistan. The evidence extracted from the data and information obtained from the validated VASA tool will surely complement healthcare professionals and policymakers in improving the practice and modifying the policy for enhancing the survival of under-five children of Karachi, Pakistan.

CHERG: Child Health Epidemiology Reference Group

VASA: Verbal autopsy/Social Autopsy

VA: Verbal Autopsy

SA: Social Autopsy

BCoD: Biological Cause of Death

SPSS: Statistical Package for Social Sciences

ICMI: Integrated Child Mortality investigation

CRVS: Civil Registration and Vital Statistics

HMIS: Health Management Information System

TPtoSCF: The Pathway to Survival Conceptual Framework

CoD: Cause of Death

MCoD: Medical Cause of Death

ACVA: Algorithm certified VA technique

PCVA: Physicians certified VA technique

C.V.: Content validation;

D.V.: Discriminant Validation;

R.A.: Reliability Analysis;

I-CVI: Item-content Validity Index;

E.V.: Eigenvalue;

C.C.: Cannical correlation;

W.L.: Wilk’s Lambda;

C.A.: Cronbach’s Alpha;

LR: Likelihood-Ratio (LR) test

WHO: World Health Organization

Ethics approval and consent to participate

The ethical approval of this study was gained from the Medical Ethics Committee, University Malaya Medical Center (201412-847) and Advance Educational Institute and Research Centre (MU/ECA/48/776). After telling the purpose and procedures of the study, all respondents’ were asked for their willingness and informed written consents were obtained. For those not willing to take part in the study, their right was respected to withdraw from the study. All responses were kept confidential and children with diarrhea during the visit were advised to seek treatment. It was made sure that in case of any emotional disturbance faced by the respondent during the data collection, the respondent will be reffered to health professionals, however, no case of emotional disturbance was noted. The study did not adversely affect the rights and welfare of the subjects and no financial compensation or provision was made.

Consent for publication

Not applicable. (This research does not include any data from an individual person.)

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The author(s) declare(s) that they have no competing interests.

Funding

The current study has no funding attached.

Authors’ constibution

MBS, CWN & WYL conceived the study idea, design and process. MBS as a Key researcher and principal investigator, involved in all aspects of this study, ranging from conceptualisation, implementation, supervision, monitoring, analysis, write up etc. MBS produced the draft of this study. HA was involved in data collection and proofreading. KA, HA and MBS undertook biostatistical analyses. All authors have read and approved the final manuscript.

Acknowledgement

We would like to thank health department, town health offices, data collectors and key personnel of the relevant communities for assisting the data collection. We would like also to extend our appreciation to the supervisors for their hard work patience in obtaining necessary information.

Moyer CA, Johnson C, Kaselitz E, Aborigo R. Using social autopsy to understand maternal, newborn, and child mortality in low-resource settings: a systematic review of the literature. Glob Health Action [Internet]. 2017;10(1):1413917. Available from: https://www.tandfonline.com/doi/full/10.1080/16549716.2017.1413917
Nichols EK, Byass P, Chandramohan D, Clark SJ, Flaxman AD, Jakob R, et al. The WHO 2016 verbal autopsy instrument: An international standard suitable for automated analysis by InterVA, InSilicoVA, and Tariff 2.0. PLoS Med [Internet]. 2018;15(1):e1002486. Available from: https://doi.org/10.1371/journal.pmed.1002486
Siddiqui MB, Ng CW, Low WY. Social Autopsy is a dire need for investigating child mortality in Pakistan. Int J Endorsing Heal Sci Res [Internet]. 2016;4(2):2–5. Available from: http://tinyurl.com/SocialAutopsyEN
Gouda HN, Flaxman AD, Brolan CE, Joshi R, Riley ID, AbouZahr C, et al. New challenges for verbal autopsy: Considering the ethical and social implications of verbal autopsy methods in routine health information systems. Social Science and Medicine. 2017.
Lisa-Marie Thomas, Lucia D’Ambruoso DB. Verbal autopsy in health policy and systems: a literature review. BMJ Glob Heal. 2018;3(2).
Kalter HD, Yaroh AG, Maina A, Koffi AK, Bensaïd K, Amouzou A, et al. Verbal/social autopsy study helps explain the lack of decrease in neonatal mortality in Niger, 2007–2010. J Glob Health [Internet]. 2016;6(1). Available from: http://www.jogh.org/documents/issue201601/jogh-06-010604.pdf
Koffi AK, Libite P, Moluh S, Wounang R, Kalter HD. Social autopsy study identifies determinants of neonatal mortality in Doume, Nguelemendouka and Abong–Mbang health districts, Eastern Region of Cameroon. J Glob Health [Internet]. 2015;5(1). Available from: http://www.jogh.org/documents/issue201501/jogh-05-010413.pdf
Koffi AK, Kalter HD, Loveth EN, Quinley J, Monehin J, Black RE. Beyond causes of death: The social determinants of mortality among children aged 1-59 months in Nigeria from 2009 to 2013. PLoS One [Internet]. 2017;12(5):e0177025. Available from: https://doi.org/10.1371/journal.pone.0177025
Nonyane BA, Kazmi N, Koffi AK, Begum N, Ahmed S, Baqui AH, et al. Factors associated with delay in care–seeking for fatal neonatal illness in the Sylhet district of Bangladesh: results from a verbal and social autopsy study. J Glob Health [Internet]. 2016;6(1). Available from: http://www.jogh.org/documents/issue201601/jogh-06-010605.pdf
Siddiqui MB, Ng CW, Low WY. To identify the non-biological causes of child mortalities in developing countries, Social Autopsy tools should be based on “The Pathway to Survival Conceptual Framework”. Int J Endorsing Heal Sci Res [Internet]. 2017;5(1):1–4. Available from: http://aeirc-edu.com/wp-content/uploads/Download-full-Paper-B.pdf
Koffi AK, Wounang RS, Nguefack F, Moluh S, Libite P, Kalter HD. Sociodemographic, behavioral, and environmental factors of child mortality in Eastern Region of Cameroon: results from a social autopsy study. J Glob Health [Internet]. 2017/04/13. 2017;7(1):10601. Available from: http://jogh.org/documents/issue201701/jogh-07-010601.pdf
Bensaïd K, Yaroh AG, Kalter HD, Koffi AK, Amouzou A, Maina A, et al. Verbal/Social Autopsy in Niger 2012–2013: A new tool for a better understanding of the neonatal and child mortality situation. J Glob Health [Internet]. 2016;6(1):10602. Available from: http://www.jogh.org/documents/issue201601/jogh-06-010602.pdf
Aggarwal AK, Kumar P, Pandit S, Kumar R. Accuracy of WHO Verbal Autopsy Tool in Determining Major Causes of Neonatal Deaths in India. PLoS One [Internet]. 2013;8(1). Available from: http://dx.doi.org/10.1371/journal.pone.0054865
Marsha DR, Sadruddin S, Fikree FF, Krishnan C, Darmstadt GL. Validation of verbal autopsy to determine the cause of 137 neonatal deaths in Karachi, Pakistan. Paediatr Perinat Epidemiol [Internet]. 2003;17(2):132–42. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-0346997048&partnerID=40&md5=be143424f4660df2ca7611dfb8170011
Midhet F. Validating the verbal autopsy questionnaire for maternal mortality in pakistan. Int J Health Sci (Qassim) [Internet]. 2008 [cited 2018 Nov 14];2(1):91–6. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3068724&tool=pmcentrez&rendertype=abstract
Mpimbaza A, Filler S, Katureebe A, Quick L, Chandramohan D, Staedke SG. Verbal autopsy: Evaluation of methods to certify causes of death in Uganda. PLoS One [Internet]. 2015;10(6):e0128801. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26086600
Murray CJL, Lozano R, Flaxman AD, Serina P, Phillips D, Stewart A, et al. Using verbal autopsy to measure causes of death: The comparative performance of existing methods. BMC Med [Internet]. 2014;12(1):5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24405531
Nausheen S, Soofi SB, Sadiq K, Habib A, Turab A, Memon Z, et al. Validation of Verbal Autopsy Tool for Ascertaining the Causes of Stillbirth. PLoS One [Internet]. 2013;8(10):e76933. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24130814
Soofi SB, Ariff S, Khan U, Turab A, Khan GN, Habib A, et al. Diagnostic accuracy of WHO verbal autopsy tool for ascertaining causes of neonatal deaths in the urban setting of Pakistan: A hospital-based prospective study. BMC Pediatr [Internet]. 2015;15(1):1–9. Available from: http://dx.doi.org/10.1186/s12887-015-0450-4
Afzal U, Yusuf A. The State of Health in Pakistan: An Overview. Lahore J Econ. 2013;18:233–47.
IGME U. Levels & Trend in Child Mortality: Estimates Developed by the Estimates developed by the UN Inter-agency Group for UN Inter-agency Group for Child Mortality Estimation Child Mortality Estimation. UNICEF; 2018.
Nisar Y Bin, Dibley MJ. Determinants of neonatal mortality in Pakistan: Secondary analysis of Pakistan Demographic and Health Survey 2006-07. BMC Public Health [Internet]. 2014;14(1). Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-84902916386&partnerID=40&md5=2a323c97d87a9ed046088bf1b54e3f1a
Siddiqui MB Low WY, Syed S, Ahmed S, Noushad S, Ali A, Fatima K, Mirza M. NCW. A Verbal/Social Autopsy (VASA) Child Mortality Inquiry to Investigate Under-Five Mortality Determinants in Slums of Karachi, Pakistan: A Mix Methods Interventional Study. Int J Endorsing Heal Sci Res. 2016;4(4):111.
Anthoine E, Moret L, Regnault A, Sbille V, Hardouin JB. Sample size used to validate a scale: A review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes [Internet]. 2014;12(1):2. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4275948/
Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Heal. 2008/01/10. 2008;31(2):180–91.
Kalter HD, Roubanatou A, Koffi A, Black RE. Direct estimates of national neonatal and child cause–specific mortality proportions in Niger by expert algorithm and physician–coded analysis of verbal autopsy interviews. J Glob Health [Internet]. 2015;5(1):10415. Available from: http://www.jogh.org/documents/forthcoming/jogh-05-010415.pdf
Bennet AE, Ritchie K. Questionnaires in medicine A guide to their design and use [Internet]. London ; New York; 1975. Available from: http://trove.nla.gov.au/work/11045163?selectedversion=NBD793226
Serina P, Riley I, Hernandez B, Flaxman AD, Praveen D, Tallo V, et al. What is the optimal recall period for verbal autopsies? Validation study based on repeat interviews in three populations. Popul Health Metr [Internet]. 2016;14(1). Available from: https://pophealthmetrics.biomedcentral.com/articles/10.1186/s12963-016-0105-1
Dikid T, Gupta M, Kaur M, Goel S, Aggarwal AK, Caravotta J. Maternal and Perinatal Death Inquiry and Response project implementation review in India. J Obstet Gynecol India [Internet]. 2013;63(2):101–7. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-84879506441&partnerID=40&md5=f25652099104439c504096bbc6f93d81
Soleman N, Chandramohan D, Shibuya K. Verbal autopsy: current practices and challenges. BullWorld Heal Organ [Internet]. 2006;84(0042-9686 (Print)):239–45. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-33644795635&partnerID=40&md5=09ffcd11e7668e66416b1919445bb9d2
Muhammad Bilal Siddiqui, Chiu Wan Ng. WYL. METHODOLOGIES OF VERBAL AUTOPSY AND SOCIAL AUTOPSY TOOLS, ADOPTED BY VASA STUDIES FOR EXPLORING UNDER-FIVE MORTALITY DETERMINANTS - A SYSTEMATIC REVIEW. Int J Endorsing Heal Sci Res. 2018;6(3):43–55.
Abouzahr C, Azimi SY, Bersales LGS, Chandramouli C, Hufana L, Khan K, et al. Strengthening civil registration and vital statistics in the Asia-Pacific region: Learning from country experiences. Asia-Pacific Popul J. 2014;29(1):39–73.

Download PDF

Version 1

posted

You are reading this latest preprint version

In-depth Validation of CHERG’S Verbal Autopsy-Social Autopsy (VASA) Tool for Ascertaining Determinants and Causes of Under-Five Child Mortalities in Karachi, Pakistan.

Status:

Version 1

Abstract

Figures

Background

Methods

The sample size

Inclusion and exclusion criteria

Ethical clearance for VASA ICMI study

Data collection for the validation study

Structure of the CHERG VASA Questionnaire

Translation & Back Translation:

Tool’s validation procedure:

Results:

Discussion:

Conclusion:

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors’ constibution

Acknowledgement

References

Supplementary Files

Status:

Version 1