In-depth Validation of CHERG’S Verbal Autopsy-Social Autopsy (VASA) Tool for Ascertaining Determinants and Causes of Under-Five Child Mortalities in Karachi, Pakistan.

Background: Globally, child mortality estimates are more clustered among the developing countries where quality data on estimates and determinants of child mortality are compromised. To achieve sustainability in reducing child mortality estimates, the integrated Verbal Autopsy and Social Autopsy (VASA) tool help in estimating prevalence and assigning medical and social causes and determinants of child survival, especially in the developing countries. A validation study of the Child Health Epidemiology Reference Group’s (CHERG) Verbal autopsy/Social Autopsy (VASA) tool has been undertaken for employing in a Karachi VASA Integrated Child Mortality Investigation-ICMI study in its urban slums. Methods: Validity and reliability of the CHERG VASA-tool were tested using face, content, discriminant validation and reliability tests on one hundred randomly selected mothers, with a recent child death event. Data were computed on SPSS (version-21) and R. Results: Testing yielded high I-CVI (>81.43%); high Cronbach's Alpha (0.843); accuracy of between 75% and 100% of the discriminants classifying births to live and stillbirths. The tool showed ICVI (>82.07% and 88.98% respectively) with high accuracy (92% and 97% respectively) for assigning biological and social causes of child deaths respectively. Conclusion: CHERG valid, reliable, and relevant conceptual non-medical health-seeking practices) child mortality cases occurring in Pakistan.

Similar to any data collection tool, it's always been a concern of whether the data collected through VA and SA are correct, reliable, and represents information what the researcher intends to collect. The tool with exceptionally high validity and reliability can only help to capture the correct data which is of public health importance. The process of validation involves testing the questionnaire on these parameters. In order to obtain best possible data, many efforts have been undertaken globally in re ning the methodology of VA (13)(14)(15)(16)(17)(18)(19). In continuation of such efforts, a few of the validation studies have been undertaken globally as well as in Pakistan to assess the capacity of VA tool in providing precise and reliable evidence on the causes of child mortality (13,14,18,19). However, no attempts have been made in validating the SA, as well as the integrated VASA tool. Keeping in view of the current status of child mortality estimates in Pakistan (which stresses upon the fact that the country is among the top ten countries which hold signi cant burden of child mortality across the globe (20)(21)(22)), a VASA ICMI study has been undertaken in Karachi. The primary aim of VASA ICMI study was to gather evidence on a comprehensive list of determinants related to under-ve mortality in the underserved locations where there is a high chance that the child mortality cases might be missed to be registered under national CRVS and HMIS. However, prior to data collection, the CHERG's VASA tool (which was chosen to be used in the ICMI) was recently validated. This validation study is one of its kind, as no VASA integrated tool has been validated so far in such detail by focusing most of the techniques for measuring validity and reliability of the tool. Also, since the CHERG's VASA integrated tool is based on a most holistic conceptual framework, i.e., "The Pathway to Survival Conceptual Framework" (TPtoSCF), we hope that the validated tool will be highly bene cial in providing extended and detailed data on the most relevant determinants and causes of under-ve mortality in Karachi, Pakistan. In this paper, we share the validation results of the CHERG VASA tool.

Methods
This validation study is a part of Karachi's VASA ICMI study (23) and objectively focuses assessing the validity and reliability estimates of the CHERG VASA questionnaire, which was later utilized in Karachi's VASA ICMI study.
The sample size Literature shows very succinct speci c guidelines (24,25) on the sample size required for pre-survey validation exercise of VASA tool, however, general guidelines suggests (25) that generally the sample size needed for any questionnaire validation should be a minimum of 20% of the study's total sample size.
The sample size of the Karachi VASA ICMI study was 400 under-ve deceased cases; therefore 20% of 400 cases, i.e., 80 cases (extra to 400 cases of the ICMI study) were recruited from the community for the questionnaire validation study. This gure was later in ated to 100 cases. Mothers of these 100 underve deceased cases (n = 100) were randomly selected from 12 selected slums across Karachi.

Inclusion and exclusion criteria
Only those mothers were included in this validation process, whose under-ve children died within past six weeks (to minimize response bias) due to any illness and those who have death certi cate (issued by health authority or health care practitioner/s) with the cause of death (CoD) mentioned on the death certi cate. The CoD mentioned on the death certi cate helped us in verifying the level of accuracy of the assigned medical-CoD (MCoD) by the physician (based on data from VA component of the VASA tool using a prior algorithm (26) as well as based on their own expertise).
Ethical clearance for VASA ICMI study The ethical approval of this study was gained from the Medical Ethics Committee, University Malaya Medical Center (201412-847) and Advance Educational Institute and Research Centre (MU/ECA/48/776).

Data collection for the validation study
Trained staff visited the included households and lled the questionnaire during direct interviews.
Collected data were shared with verbal autopsy reviewers for Physicians certi ed VA technique (PCVA) and Algorithm certi ed VA technique (ACVA). Data and CoD assigned by physicians were computed on SPSS (version 21) and R. The CoD assigned by physicians were compared with CoD mentioned on the death certi cates which served as the gold standard diagnosis of the CoD.

Structure of the CHERG VASA Questionnaire
The CHERG VASA questionnaire is a tool for collecting the data on the history of events right from the conception of the mother until the death of the child. It also captures the signs and symptoms a child encountered during the illness. The initial pre-survey CHERG VASA questionnaire consisted of VA and SA components having 14 different modules focusing the different variables ranging from demographic pro le and socioeconomic status to variables encompassing the entire continuum of care during maternal pregnancy till the death of the child.
After the section on the VASA general information, the VA component has 06 modules altogether, while the SA component has 07 modules (table-1, column-1&2). These components (VA and SA) and modules have been arranged in such an order that as the interview proceeds, the sequence of events gets unfold (table-1, column-3). To record relevant information of relevant age group (i.e., neonatal deaths, and postneonatal deaths), speci c skip patterns exists in the tool.  Translation & Back Translation: Before validation, the questionnaire was forward translated into the local language, Urdu, and then backtranslated to English based on "Brislin Back-translation" technique (27). The process involved a professional language translator and a panel of professional bilingual subject experts (paediatrician, gynaecologist, obstetrician, social scientist, and psychologist). The nal translated version was processed for the validation process.
Tool's validation procedure: As illustrated in table-2 and diagram-1, the CHERG VASA tool was evaluated on its capacity to measure and produce true and actual results, what it intends to measure-this is called the validity of the tool; and on its ability to give the same results every time it will be run elsewhere, this is called as reliability of the instrument. The complete process of this tool's validation exercise included face validation, content validation, discriminant validation, and reliability analysis is described in respective sections below.  Results: 1. Face Validation. Here, the questionnaire was reviewed and critically evaluated by the same panel (utilized during the translation exercise) on the tool's format to nd out, whether all of the sections and items be easily readable, clear and straightforward to understand and comprehend by the interviewee and respondent; and on the need of rephrasing, omitting or adding any questions. The item was phrased in such a way that the respondent would need to think before responding, thereby minimising the chances of response bias. The panellists found no need for any modi cations in terms of rephrasing of questions, omitting any option of any item, adding any option of any item, etc. The translated questionnaire which went for face validation was then named as "VASA Questionnaire-version-1". This version of the questionnaire was utilised for the content validation as well as to collect data for undertaking discriminant validation. 2. Content validation involved ve subjective reviews, each with ve homogenous experts from each professional capacity ( ve paediatricians, ve gynaecologists, ve obstetricians, ve social scientists, and ve psychologists). Each member from a panel was requested to rate every item on the VASA Questionnaire-version-1 on a 5-point likert scale based on the relevancy to the conceptual framework (1 = not relevant, 2 = somewhat relevant, 3 = relevant, 4 = quiet relevant, 5 = highly relevant). The ratings were calculated using Item-content Validity Index (I-CVI). For each item, the I-CVI was computed as the number of experts rated an item on top two boxes (i.e. quiet relevant or highly relevant), divided by the total number of experts. The level of agreement between experts Page 7/22 should be as high as possible to make the I-CVI acceptable. The data on content validation is presented in table-3 under different themes/constructs. These themes/constructs were clustered either as pre-birth themes/constructs and post-birth themes/constructs, Sect. 1 through 8 covers prebirth constructs and Sect. 9 through 16 covers during and post-birth constructs. The average rating of each section was more than 4 (on a scale of 1 to 5), and the I-CVI is more than 80% for all the sections of the VASA questionnaire. Relatively lower I-CVI was observed for "symptoms before pregnancy" (81.43%) and "Birth History of Child" (81.82%) sections.  1. Discriminant validation (construct validity). It shows the distinctiveness of different constructs within the questionnaire and helps to identify whether the items on the questionnaire that are not supposed to be related to each other are actually unrelated or not. It is one of the types of construct validity.
The "VASA Questionnaire-version-1" (after the face validation exercise) of the tool was utilised for data collection. Collected data were shared with verbal autopsy reviewers (physicians). The reviewers used their clinical reviewing expertise (process called "Physicians certi ed VA technique-PCVA") and also used an algorithm developed in prior (process called "Algorithm certi ed VA technique-ACVA") in assigning the cause of deaths. The complete data (including the CoD assigned by the physicians) were computed on SPSS (version 21) and R for the validation process. For pre-birth constructs, "birth status of the child" and for during and post-birth constructs "diseases or conditions directly leading to death" are the decisive elements with potential discriminating power. Clear clusters and grouping in response pattern on VASA elements can be seen in diagram-2 for pre-birth constructs. Therefore, valid response differences between the live birth group and stillbirth group (on the elements of each section) were used for the discriminant validity of the pre-birth constructs of the tool. Similarly, valid response differences among different diseases or conditions directly leading to the death were used for the discriminant validity of during or post-birth constructs of the tool.
For pre-birth constructs of the questionnaire, discriminant analysis was performed using birth status as a dependent variable, which was de ned in such a way that birth status was coded as "1" (one) in case of stillbirth and "0" (zero) in case of live birth, and elements of each section as discriminating or classi cation variables. The goodness of t summary of discriminant analysis models, such as eigenvalue (E.V.) of canonical discriminant function, canonical correlation (C.C.), and Wilks' Lambda (W.L.) statistics were used for the discriminant validity measures of the constructs. To analyze the discriminant validity of post-birth constructs of VASA questionnaire, multinomial logistic regression was performed using "disease or condition directly leading to death" as dependent variables and elements of each section of the questionnaire as explanatory variables. Likelihood-Ratio (LR) test and Pseudo R-Square measures (Cox and Snell, Nagelkerke, and McFadden) were used as discriminant validity measures of the constructs.
Out of 100 cases, we had 33 stillbirths and 67 live birth cases. For the analysis, unequal prior probabilities of stillbirth and live birth were assumed and assigned prior probabilities as per the group size in data. Backward variable selection method was used to compute the discriminant analysis model. It is vital to comment here at this point that the variables which are not included in the nal model do not necessarily mean they are not important, but it means these variable(s) does not provide any additional statistical evidence towards the classi cation of live and stillbirth.
The goodness of t summary of discriminant analysis models of each of the pre-birth constructs of VASA questionnaire is presented in table-4. The eigenvalue (E.V.) of canonical discriminant function is the ratio of between groups sum of squares to within groups sum of squares. A high value of eigenvalue indicates that the canonical function explains su ciently higher variation in data. Similarly, canonical correlation (C.C.) coe cient shows the predictive ability of canonical discriminant function. The value of C.C. ranges from 0 to 1, with a value closer to 1 shows the strong predictive ability of canonical discriminant function. Third goodness of t statistics i.e. Wilks' Lambda (ranging from 0 to 1), was calculated to be the proportion of the total variance in the discriminant scores not explained by differences among the groups. A small value of W.L. and statistically signi cant (p-value of < 0.05) shows the good t of the discriminant model.   affect the quality and reliability of data. This is most likely because the respondent needs to recall the events and happenings of different events that occurred before and around the death even. It is highly recommended that VA interviews should be undertaken somewhere between 01 months till 06 months provided the family have come out of the mourning period (28). Conducting interviews within the mourning period may cause distress and in uence the respondent's disposition and ability to engage in VA interview (28). P Serina et al. (28), suggests that a long recall period may limit the respondent's ability in recalling and recollecting pertinent facts. Events (or symptoms) with extraordinary severity (or implications) remain in the recall for a much more extended period compared to those with mild to moderate severity (28). Likewise all the other studies (14,18,19,29), our study undertook interviews within the six (06) weeks after the mortality event. In our sample size, none of the respondents had any di culty in recalling the events.
3. Tool's accuracy: The capacity of any questionnaire to correctly assign the proper CoD to any mortality (due to any speci c reason) with a degree of accuracy is called accuracy. When dealing with individual-level data, the diagnostic accuracy of VA is considered to be satisfactory when the speci city and sensitivity of the questionnaire are a minimum of 90% (30). However, at a much larger, i.e. national level, if the validation attempt of VA shows sensitivity and speci city of at least 50% and 90% respectively, the tool is usually considered to be having acceptable diagnostic accuracy (30). Our validation study de nes speci city as the ability of the VA section of CHERG's VASA questionnaire to accurately and correctly assign the medical cause of death compared to the assigned CoD on the medical certi cate as well as the based on the signs and symptoms. The closeness of Medical CoD between the one assigned in the death certi cate and the one assigned through VASA inquiry was used to measure the validity of the tool. Our study showed accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively. In comparison, the accuracy of the WHO's VA tool validated in Aggarwals' study (13) in identifying different neonatal causes of mortalities was found to up to between 78-92% (except for birth asphyxia-16%); however, the kappa statistics was moderate from 0.46 to 0.55. Similarly, in Soo et al. (19) the WHO VA tool showed the sensitivity of more than 83.5% for diagnosing different causes of neonatal mortality (except congenital malformation, which was 57%), while the speci city of all the major causes of neonatal deaths was found to be more than 90%. Marsha et al.(14) found the sensitivity of the process of assigning the cause of death ranging from 39% (in diagnosing infection) to 90% (in diagnosing causes related to too early/too small syndrome) and speci city ranging from 67% (in diagnosing infection) and 99% (in diagnosing Neonatal Tetanus).

Validation of SA and VASA tools
The literature shows a lack of evidence on validation studies for SA and integrated VASA tool for underve years children. More speci cally, global as well as local (Pakistan speci c) literature is lacking on any attempt to endorse the level of validity and and reliability of CHERG VASA questionnaire in diagnosing mortalities of under-ve children (born as still and live births) and reported as deceased in a communitybased household survey with events reported as full illness-related history. Although the CHERG VASA tool has been designed by experts, still its validation for developing country like Pakistan (where the huge estimates of under-ve mortality lies) is technically required, ensuring the availability of a VASA tool that may give the valid and reliable data pertaining to child mortality determinants speci cally operating in Pakistan. Such tool is highly required to overcome the data shortage related issues in the country, where the birth and death registration are highly compromised (3,31,32). The existing data is also very limited and with questionable validity and accuracy (3,31,32). Utilization of this validated tool in identifying mortality related determinants and in assigning medical and social causes of child deaths, especially of those deaths which have been missed from the National Civil Registration and Vital Statistics will surely help us in overcoming data related issues pertaining to child mortalities.

Strengths of our validation study
Our study is rst in its kind as it validates the CHERG's integrated VASA tool, which is based on the most holistic conceptual framework, i.e. TPtoSCF (that addresses the barriers and limitations involved in accessing health care services), thereby recording information required to develop policy in preventing under-ve death incidents and improving the child survival estimates in developing countries (like Pakistan) where an unknown, but large number of child mortality events have known to be missed and their medical causes have not been assigned.
Moreover, this study is highly important as it attempted a detailed validation of CHERG's integrated VASA tool by incorporating rigorous methodology and utilising almost all the core methods suggested for identifying the validity and reliability evidence of any questionnaire. The validation exercise yielded high I-CVI (> 81.43%), with Cronbach's Alpha reliability statistics is 0.843 for "Demographic and social constructs" and highest of 0.973 for "Preventive Measures between childbirth and illness"; accuracy of between 81% (for "Care seeking behavior for symptoms of last trimester") and 100% of the discriminants classifying births to live and stillbirths. The tool showed accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively.
Although different VA questionnaires have been validated in Pakistan, however, some of them are speci cally designed for the neonatal population (19), while others are for stillbirths (18). The CHERG's VASA integrated questionnaire encompasses three age groups and has not been validated as for the Pakistani population.
An electronic CAPI (Computer Assisted Personal Interview) notebook format for data collection with skip patterns using CSPro software application has also been developed for this validated tool which was utilized for data collection during the VASA ICMI survey. The primary purpose for transforming the lay paper-based form to notebook application is to minimize the data entry errors.

Limitations of the study
Although we undertook our validation study on the recommendations that sample size for validation should be at least 20% of the sample size chosen for survey, still our sample size is smaller.

Recommendations
Future research should be geared to undertake the validation of different VA and SA tools (as separate tools). This should be coupled with the efforts on enhanced research on the e cacy of SA tool and to improve its capacity and accuracy for identifying social determinants pertaining to under-ve mortality. Moreover, there is a strong need to have speci c recommendations for estimated sample size explicitly required for the validation process of the VASA tool. This would help future researches to undertake analyses based on standardized recommendations. Additionally, it would be much better to conduct quantitative and qualitative assessment on the respondents point of view and their feedback towards the tool.

Conclusion:
Based on our results, we con rm that the CHERG VASA integrated questionnaire is valid, reliable, and relevant to the conceptual frameworks. The modi ed CHERG VASA tool can be utilized for establishing medical and social causes of Under-ve child deaths in Karachi, Pakistan and is one of the assets for Pakistan's child health policy with the potential to record relevant data and assign causes of child mortality cases occurring in Pakistan. The evidence extracted from the data and information obtained from the validated VASA tool will surely complement healthcare professionals and policymakers in improving the practice and modifying the policy for enhancing the survival of under-ve children of After telling the purpose and procedures of the study, all respondents' were asked for their willingness and informed written consents were obtained. For those not willing to take part in the study, their right was respected to withdraw from the study. All responses were kept con dential and children with diarrhea during the visit were advised to seek treatment. It was made sure that in case of any emotional disturbance faced by the respondent during the data collection, the respondent will be reffered to health professionals, however, no case of emotional disturbance was noted. The study did not adversely affect the rights and welfare of the subjects and no nancial compensation or provision was made.

Consent for publication
Not applicable. (This research does not include any data from an individual person.)

Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.