Validity of a visual analogue scale to measure and value perceived level of sanitation – evidence from Ghana and Mozambique

doi:10.21203/rs.3.rs-3752636/v1

Download PDF

Research Article

Validity of a visual analogue scale to measure and value perceived level of sanitation – evidence from Ghana and Mozambique

https://doi.org/10.21203/rs.3.rs-3752636/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Two billion people globally lack access to a basic toilet, and sanitation is a critical determinant of health and wellbeing. Evaluations of sanitation programmes typically measure disease or behaviour, and visual analogue scales (VAS) have not been used to measure users’ feelings about their level of sanitation. In this study, we assess the validity of a sanitation VAS numbered 0–10, with end-anchors best and worst imaginable sanitation. In Kumasi, Ghana, we surveyed 291 participants before and after uptake of a container-based sanitation service. In Maputo, Mozambique, we surveyed 424 participants from treatment groups of a prior trial. We assessed construct validity by testing hypothesised associations between VAS scores and toilet characteristics, and by respondents valuing three hypothetical sanitation states. We assessed responsiveness by comparing VAS with/without interventions. There was evidence (p < 0.05) for 60% of hypothesised associations in Ghana, and 100% in Mozambique. For responsiveness, there was a 3.4 point increase (2.1 SD) in VAS 10 weeks post- intervention in Ghana, and a 2.9 point difference (1.3 SD) in Mozambique. In valuation exercises, the mean was higher (p < 0.001) for the objectively better state. The sanitation VAS could be useful in economic evaluation, in valuing sanitation states or levels of service, to identify which improvements achieve quality of life gains most efficiently.

Health Economics & Outcomes Research

economic evaluation

visual analogue scale

sanitation

validity

Evaluations of sanitation programmes often focus on disease outcomes (Wolf et al., 2022) and/or behaviour change (Garn et al., 2017). Impact evaluations are beginning to measure sanitation-specific dimensions of quality of life (Freeman et al., 2022; Ross et al., 2022). However, few studies have assessed participants’ overall evaluation of their sanitation status or satisfaction (Pickering et al., 2015; Tumwebaze et al., 2013), and none have assessed validity of the measures used. An individual’s subjective perception of their level of sanitation overall could be complementary to more specific outcome meaures, but also provides information in its own right. Such measures might questions ask about “your level of sanitation” or “satisfaction with your toilet” on some scale. This requires the respondent to rapidly and unconsciously integrate multiple quality of life factors important to them, such as privacy, absence of smell, social status, safety, disease risk, etc.

A visual analogue scale (VAS) is a line with a rating scale on which respondents report an outcome, providing a measurable value function for ranking preferences (Dyer & Sarin, 1979). Intervals between placements on the line reflect differences in preference (Torrance, 1976). VAS scores represent strength of preference under certainty, as a value function, as opposed to a utility function which represents preferences under uncertainty (Keeney & Raiffa, 1975; Robinson et al., 2001).

VAS have been used in health economic evaluation since the 1970s (Patrick et al., 1973). They have varied in respect of the labels for the ends of the line (e.g. full health vs. death), vertical or horizontal presentation, and presence or absence of scale marks and numbers (Torrance et al., 2001). Health economists most commonly use VASs to measure a respondent’s subjective health status or to elicit a respondent’s valuation of hypothetical health states (Parkin & Devlin, 2006). The EuroQoL VAS (EQ-VAS), reproduced in Supplementary Material A, is a health status VAS used in over 40,000 studies to date (Cheng et al., 2021; Devlin et al., 2020), providing a measure of overall health-related quality of life. VAS have also been used to specific dimensions of health-related quality of life, such as pain (McCarthy et al., 2005).

The main advantages of VAS are their quick, inexpensive administration and amenability to self-completion (Torrance et al., 2001), whether in paper, digital, or voice recognition surveys (Lundy & Coons, 2011). VASs have two main disadvantages when used in valuation (Torrance et al., 2001). First, context bias, whereby the VAS score for a state depends on the number of better and worse states presented at the same time (Bleichrodt & Johannesson, 1997). Second, end-aversion bias, whereby some respondents appear reluctant to select ratings near the end-anchors (Streiner et al., 2015). However, adjustment for these properties is possible (Parkin & Devlin, 2006). A systematic review of the measurement properties of the EQ-VAS found it to have “sufficient” construct validity in most populations, but “sufficient” test–retest reliability in fewer populations (Cheng et al., 2021).

In this study, we aim to assess the validity of a novel sanitation VAS for use in impact and economic evaluations, in the context of urban sanitation interventions in Mozambique and Ghana. In doing so we address the need for validated and reliable sanitation measures focused on quality of life beyond disease, which can be simply and consistently deployed in studies and routine monitoring. When researchers measure outcomes in impact evaluations and economic evaluations, they need to be confident that they are truly measuring what they think they are measuring. Assessing validity is one means of providing that confidence or demonstrating that it is misplaced.

We use data from two previous evaluations of urban sanitation interventions in Ghana (Tidwell et al., 2022) and Mozambique (Ross et al., 2022).

Maputo setting and intervention

Maputo City has a population of 1.1 million, with the majority living in basic settlements with unpaved roads. Our study site comprises low-income neighbourhoods in a 10 km² area of the Nhlamankulu urban district, where the poorest people live in informally walled ‘compounds’ with many households in small single-storey dwellings sharing the same toilet. Low-quality pit latrines are common, often with squatting slabs made of wood or tyres, and no water seal providing a barrier to smells and flies. Privacy can be a challenge since latrine walls are often made with scrap corrugated iron or plastic sheeting. The study design was observational. In 2019, 424 participants recruited from intervention and control compounds of a prior non-randomised trial of a toilet subsidy programme (ClinicalTrials.gov, NCT02362932). Two people aged 18 + were recruited per compound (one man, one woman) from different households. Intervention compounds were provided with a subsidised pour-flush toilet with a water seal. They discharged to a septic tank with soakaway, had a concrete superstructure, and had metal doors lockable from the inside, all of which make them objectively higher-quality than control toilets. Compound inhabitants paid a 10–15% capital contribution. The study setting, intervention, sampling strategy, and other methods are described in more detail elsewhere (Ross et al., 2022).

Ghana setting and intervention

Kumasi is Ghana’s second-largest city, with a population of 2.7 million. Independently operated pay-per-use public toilets, of which there are 400 around the city, are the primary sanitation facility for 36% of the city’s population (Ghana Statistical Service, 2013). Two-thirds of participants in a recent study in Kumasi had to walk at least 400m to reach the nearest public toilet (Gaisie et al., 2018). Our study evaluated container-based sanitation (CBS) services provided by Clean Team Ghana (CTG), which rents out high-quality plastic toilets with a sealable internal waste container collected and replaced weekly. The design was a before-after enrolment study of CTG customers aged 18+, with participants surveyed shortly before installation of the toilet and 10 weeks afterwards. The study design focused on self-selecting customers, so no control group was used. Participants lived in 10 different Metropolitan and Municipal Assemblies (districts) within the Greater Kumasi Metropolitan Area. Housing was mixed, with some multi-storey tenement housing and some single-storey dwellings. Of the 404 people recruited at baseline in 2019, only 292 users completed both pre-CTG and post-CTG surveys. The study setting, intervention, sampling strategy, and other methods are described in more detail elsewhere (Tidwell et al., 2022).

Study design and VAS

We use datasets from the above studies to explore the performance of a sanitation VAS (Fig. 1), assessing its construct validity, responsiveness, and convergent validity (explained further below). In Ghana, of 292 individuals interviewed at endline, 291 had VAS data at baseline (used for construct validity analyses) and 280 had VAS data for endline as well (used for responsiveness analyses). In Mozambique, all n = 424 individuals had VAS data. The sanitation VAS comprises a horizontal line, with 11 scale marks and numbers at every point from 0–10. Emojis were included to aid participants’ interpretation. This decision was informed by a pain VAS (Hawker et al., 2011) and by discussions during piloting in Mozambique, where fieldworkers agreed that it was more appropriate to have grayscale emojis than to reflect typical Mozambican skin tones or have yellow emojis. The end-anchor for zero is labelled ‘worst imaginable sanitation’ and 10 ‘best imaginable sanitation’, adapted from the EQ-VAS (Cheng et al., 2021). In each setting, end-anchors and guidance were depicted and explained in local languages (Portuguese in Maputo, Twi in Kumasi). We asked participants to rate on the scale how they felt about their ‘level of sanitation today’.

Table 1

Hypothesised associations of VAS scores with toilet quality variables
	Kumasi	Mozambique	Direction of hypothesised association (and rationale)
Hypothesised to be associated with VAS score
Floor/slab	Toilet has ceramic pan/floor	Toilet floor is manufactured material*	Positive (more modern and easier to keep clean compared to lower-quality floors)
Water seal	Toilet has water seal	n/a (100% collinearity with intervention)	Positive (keep out smells and flies compared to direct-drop pit latrines)
Roof	n/a (96% have)**	Toilet roof is manufactured material*	Positive (stops people looking in from above and prevents rain entering)
Lock	n/a (97% have) **	Toilet locks from the inside	Positive (stops others entering, by mistake or on purpose)
Cleanliness	Toilet pan is not visibly dirty with faeces	Enumerator does not smell faeces	Positive (less disgusting to use)
Solid waste	n/a (not collected)	No solid waste observed around floor	Positive (less disgusting to use). nb. solid waste referred to waste other than anal cleansing materials
On-compound	Toilet is on-compound	n/a (100% on-compound)	Positive (easier and quicker to access, with more safety and less worry)
Handwashing	Handwashing facility near toilet	n/a (not collected)	Positive (easier to feel clean after using the toilet)
Negative controls (hypothesised not to be associated with VAS score)
Years in dwelling	Respondent years lived in that dwelling		n/a (no obvious rationale for association with VAS score)
Education	Respondent completed primary education
Partner	Respondent has a partner

*manufactured materials included cement, zinc sheets, tiles, etc. **We only included variables if < 85% of the sample was in a category, to ensure a minimum of statistical power. Ideally we would have tested for association with sharing the toilet, but almost all respondents in both samples shared their toilet with other households (Table 2).

Construct validity

The assessment of construct validity focuses on how well a measure reflects a concept (construct) that is not directly measurable (Fayers & Machin, 2015). To assess construct validity, we pre-specified hypotheses (Table 1) about the presence of associations between VAS scores and a set of toilet characteristics, drawing on the literature on motives for sanitation behaviours and wellbeing (Novotný et al., 2018; Sclar et al., 2018). We also included some negative controls (Arnold & Ercumen, 2016), hypothesised not to be associated with VAS score. In Ghana, we used the baseline dataset before the CBS toilet was delivered, since at endline all were using CBS. We tested hypotheses using generalised linear mixed models (GLMM) in Stata 17, by regressing on VAS score including in turn each of the variables indicated in Table 1, per country. We also explored the consequences of accounting for covariance between the toilet characteristics, by regressing on all variables concurrently. In both countries, models are two-level GLMMs with standard errors clustered at the district level (Ghana) or compound level (Mozambique). In Ghana we cluster at the district level because there may be neighbourhood-level factors (e.g. security, local public toilet quality or crowding) which influence how people perceive their level of sanitation. In Mozambique, all respondents lived in a small area of a single district, but the two people recruited per compound (from different households) usually shared the same toilet.

Convergent validity

Convergent validity explores whether two measures aiming to capture similar constructs are correlated (Fayers & Machin, 2015). We assessed convergent validity by correlation (Pearson's r) between VAS scores and an index of sanitation-related quality of life (SanQoL). The SanQoL index was developed in Mozambique (Ross, Greco, et al., 2021) and has now been used in several other countries. Its questions (Supplementary Material B) measure the respondent’s degree of achievement of five attributes: privacy, disgust, shame, disease, and safety related to sanitation (Ross, Cumming, et al., 2021). Higher SanQoL index values represent better quality of life, with weighting of the five attributes arrived at via preference elicitation (Ross, Greco, et al., 2021). We hypothesised that the correlation between VAS and SanQoL would be positive and greater than 0.5, because they are capturing similar concepts, and following norms for the EQ-VAS (Whynes et al., 2008). For the Mozambique dataset the sample design (two respondents per compound) permitted investigation of the convergence of VAS scores between users of the same toilet. This was achieved based on inter-rater reliability methods using the intracluster correlation coefficient (ICC) with a one-way random effects model (Koo & Li, 2016). The interpretation of this ICC is “fair” (0.40–0.59), “good” (0.60–0.74), or “excellent” (> 0.75) (Cicchetti, 1994). Our hypothesis was that VAS scores would be strongly positively correlated (> 0.5), but not completely correlated (< 0.9), because two people may experience the same toilet differently.

Responsiveness

Responsiveness is the ability of a measure to detect changes over time in the targeted construct (Fayers & Machin, 2015). We assessed responsiveness in Ghana by assessing the difference in VAS scores before/after the intervention. We assessed the similar concept of “known-groups validity” in Mozambique, comparing intervention and control. Both analyses were done using GLMMs, adjusting for sex, being aged over 60, and an asset-based wealth index (rationale discussed in Ross et al., (2022)). In Ghana, the model is a three-level GLMM with random effects at the individual and district level, and standard errors clustered at the district level. In Mozambique, the model is a two-level GLMM with standard errors clustered at the compound level. We report the effect size in standard deviations (SD), a commonly-used measure of responsiveness (Fayers & Machin, 2015).

Validity in valuation

In Ghana, a valuation exercise included in the study at baseline provided the opportunity to investigate another aspect of construct validity of the VAS. We developed hypothetical “sanitation states” as combinations of SanQoL attribute levels (example in Fig. 2). At this point in the questionnaire, respondents had already answered SanQoL questions and VAS for themselves, so were familiar with the concepts. On the cards, each attribute was visualised with an emoji, and each attribute level (e.g. always, sometimes, rarely, never) visualised by a number of happy/unhappy emojis. Fieldworkers first explained the SanQoL card, then asked the respondent to value the state on the VAS. Two further states were also valued (Supplementary Material C). We hypothesised that the SanQoL state which is objectively better than the other two would have a higher mean VAS score in paired t-tests.

Ethics

The Mozambique study received prior approval from the Comité Nacional de Bioética para a Saúde (ref: IRB00002657) at the Ministry of Health in Mozambique. The Ghana study received prior approval from the Committee on Human Research, Publications and Ethics, Kwame Nkrumah University of Science and Technology (ref: CHRPE/AP/317/19). The protocol for present study was reviewed by the MSc Research Ethics Committee at LSHTM, which concluded that additional ethics approval was not required (ref: 27732). Informed, written consent was obtained from all participants in both countries.

Table 2

Sample characteristics for datasets used in construct validity analyses
	Ghana at baseline (n = 291)	Mozambique (n = 424)
Respondent demographic characteristics
Respondent is female	215 (77%)	220 (52%)
Respondent mean age	44.1 (12.7)	39.9 (15.3)
Aged 18–29	14 (5%)	126 (30%)
Aged 30–44	147 (51%)	155 (37%)
Aged 45–59	84 (29%)	88 (21%)
Aged 60+	46 (16%)	55 (13%)
Household size	3.2 (1.7)	5.1 (3.0)
Completed primary school or above	192 (67%)	268 (63%)
Piped water on-premises	63 (23%)	416 (98%)
Sanitation characteristics
Type of toilet
Flush or pour-flush toilet	223 (83%)	222 (52%)
Pit latrine	47 (17%)	202 (48%)
Nature of sharing
Not shared with other households	13 (5%)	47 (11%)
Shared but not public toilet	62 (23%)	377 (89%)
Public toilet	195 (72%)	0 (0%)
Toilet is on-plot	42 (16%)	416 (98%)
Toilet has solid walls	205 (74%)	275 (65%)
Toilet has inside lock	182 (66%)	187 (44%)

Data are n (%) for categorical variables and mean (SD) for numerical variables. Percentages for categorical variables are % of those with data for that variable.

Sample characteristics

In Ghana, most respondents were female (77%) while in Mozambique about half were (52%)(Table 2). Ghanaian respondents were slightly older on average (44.1) than Mozambican repsondents (39.9). The average household size was higher in Mozambique (5.1) than in Ghana (3.2). Almost all Mozambican participants had on-premises piped water (98%) while only 23% of Ghanaian participants did. In Mozambique, participants were equally split between users of pit latrines and users of pour-flush toilets, and almost all were using a toilet which was shared with other households (89%) and/or which was on-plot (98%). In Ghana, most participants used flush or pour-flush (83%) toilets at baseline, and mostly pay-per-use public toilets (72%). We do not report statistical tests comparing the two samples in Table 2 because differences between them are not relevant to our analyses, and validity is best assessed across diverse populations. Differences between groups compared in Mozambique are reported and discussed elsewhere (Ross et al., 2022).

Validity

When hypotheses for the Ghana sample were assessed individually (Table 3), there was evidence at the 10% level for 80% of posited associations between VAS scores and toilet characteristics (60% at 5% level). For the Mozambique sample, there was evidence at the 10% level for 100% of hypotheses (100% at 5% level). When assessed concurrently, there was evidence at the 10% level for 80% of hypotheses in Ghana (40% at 5% level) and 100% in Mozambique (80% at 5% level). In neither country was any of the negative controls associated with VAS scores at the 10% level.

Table 3

P-values on coefficients for hypothesised associations in GLMM regressions, individually and concurrently
	Ghana		Mozambique
	individual	concurrent	individual	concurrent
Hypothesised to be associated with VAS score
Floor/slab	0.061*	0.646	< 0.001***	0.001***
Water seal	< 0.001***	0.016**	n/a	n/a
Roof	n/a	n/a	< 0.001***	0.075*
Lock	n/a	n/a	< 0.001***	0.011**
Cleanliness	0.010**	0.057*	< 0.001***	0.02**
Solid waste	n/a	n/a	< 0.001***	0.015**
On-compound	0.223	0.069*	n/a	n/a
Handwashing	< 0.001***	< 0.001***	n/a	n/a
Negative controls
Years in dwelling	0.184	n/a	0.720	n/a
Education	0.300	n/a	0.482	n/a
Partner	0.112	n/a	0.839	n/a

All associations were in the hypothesised direction. Individual models regressed on VAS scores and the indicated variable. Concurrent models regressed on all hypothesised variables at once. Models are two-level GLMMs with standard errors clustered at the district level (Ghana) or compound level (Mozambique).*, **, *** indicate significance at the 10, 5 and 1 percent level. Pairwise correlations between variables are tabulated in Supplementary Material C. Detailed regression output for concurrent models is in Supplementary Material E, including coefficient values.

Correlation between VAS scores and the sanitation-related quality of life index was 0.51 (p < 0.001) in Mozambique and 0.70 in Ghana (p < 0.001), in line with hypotheses and similar to correlations ideintified between the EQ-VAS and the EuroQoL five-dimension index (EQ-5D)(Whynes et al., 2008). Distributions of VAS scores and SanQoL index values with/without intervention are in Supplementary Material F. In Mozambique, the ICC was 0.78 for convergence of VAS between two respondents (from different households) using the same toilet on the same compound, indicating substantial but not complete correlation as hypothesised.

Table 4

Evidence for responsiveness between groups with/without an intervention in GLMM regressions
		Ghana (n = 280)		Mozambique (n = 424)
		without intervention	with intervention	without intervention	with intervention
Sample size		280 (before)	280 (after)	202 (control)	222 (int'n)
Mean VAS score	Mean	5.1	8.6	4.1	7.0
Mean VAS score	(s.e.)	(0.1)	(0.1)	(0.2)	(0.1)
Unadjusted models	Unadjusted diff. (95% CI)	3.4*** (3.2–3.6)		2.9*** (2.4–3.4)
Unadjusted models	p-value	< 0.001		< 0.001
Adjusted models	Adjusted diff. (95% CI)	3.4*** (3.2–3.6)		2.9*** (2.4–3.4)
	p-value	< 0.001		< 0.001
	Effect size (Cohen’s d)	2.1 SD		1.3 SD

In Ghana, the model is a three-level GLMM with random effects at the individual and district level, with standard errors clustered at the district level. In Mozambique, the model is a two-level GLMM with standard errors clustered at the compound level. Adjusted models include gender, aged 60+, and wealth index as covariates. *, **, *** indicate significance at the 10, 5 and 1 percent level. Detailed regression output is in Supplementary Material F.

Responsiveness

There was good evidence for responsiveness, with a 3.4 point increase in VAS scores (95% CI: 3.2–3.6) in Ghana after the intervention. In Mozambique there was a 2.9 point difference (95% CI: 2.4–3.4) in VAS scores between the intervention and control groups. Distributions of VAS scores by treatment group clarify the pattern of responses driving these results (Fig. 3). Effect sizes were large at 2.1 SD in Ghana and 1.3 SD in Mozambique. Limitations of study designs for causal inference are explored in the discussion section – the important result here is the responsiveness of VAS scores, irrespective of bias.

Valuation

Participants valued three states (Table 5), of which State 3 is objectively better than the other two. For none of the five attributes is State 3 worse than State 1, and it is preferable to State 2 on all but one attribute. In line with our hypotheses, valuations for State 3 were significantly higher (p < 0.001) in paired t-tests than for States 1 and 2. For State 1 and State 2, which are relatively similar in terms of overall attribute levels, the 95% CIs of means overlap.

Table 5: mean VAS valuations of three SanQoL states (n=291)

	State 1	State 2	State 3
Disgust	rarely	Never	sometimes
Disease	never	Never	sometimes
Privacy	rarely	sometimes	always
Shame	never	Never	sometimes
Safety	rarely	always	rarely
mean	4.2	4.4	5.5
(95% CI)	(4.1-4.3)	(4.2 - 4.5)	(5.1 - 5.8)

A sanitation state is a combination of SanQoL attribute levels, where “never” for all five attributes is the worst-possible state, and “always” for all five is the best-possible (Supplementary Material B).

In this study, we have evaluated different aspects of validity of a sanitation visual analogue scale (VAS) on which respondents indicate how they feel about their ‘level of sanitation today’ on a 0–10 scale with emojis. There was evidence at the 5% level for 60–100% of hypothesised associations between VAS scores and toilet characteristics when assessed individually, and 40–80% when assessed concurrently. There was good evidence for responsiveness, with effect sizes on VAS scores of 1.3–2.1 SD associated with sanitation interventions which included objective level of service substantially. We observed convergence between VAS scores and SanQoL index values, and between VAS scores of two respondents using the same toilet. To our knowledge, this study represents the first exploration of the validity of a sanitation VAS, and one of the few studies of the validity of sanitation measures focused on quality of life.

It was surprising that the variable for “toilet is on compound” (as opposed to having to leave the compound) was not significant as an individual association in Ghana (Table 3). However, that variable was negatively correlated with “ceramic pan/floor” and “water seal” variables (Supplementary Material D), suggesting that on-compound toilets were more likely to be low-quality, which potentially drives this result. It also explains why someone who already has an on-compound toilet might want to invest in container-based sanitation as an upgrade. Accounting for these and other variables in the concurrent regression, the on-compound variable becomes significant at the 10% level with a positive coefficient, as hypothesised (Supplementary Material E).

The benefits of sanitation beyond infectious disease are likely to underpin household willingness to pay for sanitation improvements. These benefits are diverse (Novotný et al., 2018), and different outcomes are important to different people. Global measures such as VAS can capture an overall picture of how people feel about their situation with respect to an outcome (Parkin & Devlin, 2006). Measuring toilet characteristics objectively has often been the focus of efforts to assess sanitation quality (Schelbert et al., 2020; Tidwell et al., 2018). Measures like the sanitation VAS focus on the subjective experience of the individual. This is important because sanitation interventions might improve infrastructure and/or behaviours, but with no impact on quality of life outcomes.

The sanitation VAS has practical applications in impact evaluation. Our responsiveness analyses illustrate how different sanitation programmes might be compared in respect of their effectiveness for self-perceived level of sanitation. VAS may not be an obvious candidate for a primary outcome in an impact evaluation, but it could be a useful addition to the armoury of secondary outcomes to give a more rounded picture of comparative effectiveness. In the broader Mozambique study, for example, the evaluated intervention was found to have no effect on health outcomes such as diarrhoea or stunting after 24 months (Knee et al., 2021). However, it did have a substantial impact on VAS (Table 4), providing complementary information about different dimensions of effectiveness important to both decision-makers and toilet users. The simplicity of the VAS and its amenability to self-completion could make it particularly useful in SMS or WhatsApp-based surveys.

The sanitation VAS also has applications in economic evaluation and in understanding preferences (Ross, 2022), since it provides a measurable value function whereby intervals between placements reflect differences in preference (Torrance, 1976). These properties are useful in understanding the relative value of states of the world. We present results of such an application in valuing SanQoL states (Table 5). However, VAS could equally be used to understand the relative value of specific sanitation service options (e.g. pit latrine versus pour-flush toilet) or other service characteristics (e.g. public toilet at 5-minute round-trip, versus 10- or 15-minute). Other analyses might the relative contribution of improvements in toilet characteristics to increases in VAS scores, to identify where the largest incremental gains might be made at low cost. In construct validity analyses, what is important is not so much the relative size of coefficients (reported in Supplementary Material E), but rather in the fact that associations are present in the hypothesised direction and significant (Reeve et al., 2013).

Comparing SanQoL-5 scores and VAS scores amongst the same individuals, a slight end-state aversion is seen (Parkin & Devlin, 2006). For example, respondents tend to avoid selecting a score of 10 (Fig. 3), even when they have the highest possible levels for all SanQoL questions. In the "with intervention" distributions for both Ghana and Mozambique, the modal VAS score is 8 whereas the modal SanQoL index value is 1.0 (Supplementary Material F), which reflects the pattern observed between EQ-VAS and EQ-5D (Whynes et al., 2008). The two outcomes measure similar but slightly different things, so are complementary.

Given the evidence in support of validity we have presented, and the relative simplicity and directness of the question, the sanitation VAS is likely to be appropriate for general use. However the assessment of validity is an ongoing process and, in particular, an exploration of people’s cognitive processes when interpreting and responding to the sanitation VAS would be helpful, e.g. a think-aloud study (Ernstsson et al., 2020). Such an assessment could also explore alternative formulations of the question and end-anchors. In future studies we recommend a 0-100 scale to allow more granularity, with emojis applied only at end-anchors to avoid the risk of the emojis being used as a heuristic to ignore the numeric scale. The proposed new formulation, depicted in Supplementary Material H, is closer to a truly continuous analogue scale (Price et al., 2012).

Our study does have some limitations. First, the evidence we have provided is from two quite specific contexts alongside evaluations of interventions. It would be important to understand the properties of the sanitation VAS in the general population, especially rural areas. The specificity of these datasets is also a limitation to the construct validity assessments, which would ideally be repeated in a sample of people using a variety of different types of sanitation. This would also allow more hypotheses to be tested, e.g. the role of sharing. Second, the effect sizes in the responsiveness analyses should be compared with caution, since the two samples we draw on differ in important characteristics (Table 2), and the study designs and interventions are also different. Notably, the Ghanaian sample comprised self-selecting individuals who had signed up to the CTG service whereas the Mozambican sample was drawn from trial-enrolled compounds. The Ghana study has no control group and so is likely to have greater risk of bias than the Mozambique study, and further limitations regarding causal inference are discussed in the respective parent study papers (Ross et al., 2022; Tidwell et al., 2022). However, the focus of the present study is assessing responsiveness of the VAS rather than comparing effect sizes between interventions, and our study design is appropriate for this purpose. Third, only certain aspects of validity and reliability could be tested given the data available in the underlying studies. Future studies would ideally investigate test-retest reliability, namely whether someone’s VAS score is sufficiently stable in the short-term, e.g. comparing two measurements 7–14 days apart (Polit, 2014), which was not possible in these studies.

There are also limitations to the VAS itself. First, the fact that people incorporate different subjective weightings of sanitation-related outcomes into their VAS response is a limitation to the extent that two people’s responses may be based on different things. One person may base their response primarily on their recent experience of disgust, while another might base it primarily on privacy. However, this is also a strength in that it allows people to weigh up what is important to them, in the same way as questions about life satisfaction and happiness do (Welsch, 2006). Second, an individual’s evaluation of their current sanitation status is affected by “adaptation” to previous experience of toilets (Nussbaum, 2001). Most people globally, though not all, are likely to have experienced “full health” for some part of their life. However, fewer people will have experienced the best level of sanitation they can imagine (Fig. 1), and people vary in respect of their imagination and their prior experience. An evicted tenant who has moved from using a high-quality toilet to a low-quality one (in their new home) may attribute a lower VAS score to their current status than someone who has only ever used a low-quality toilet, due to the relative deprivation compared to their prior experience.

When evaluating a policy’s effectiveness on an outcome, we need to be confident that our measurement of that outcome is truly capturing what we think it is. This is especially true if we are asking policy-makers to make decisions based on findings. For example, estimated prevalence of diarrhoea (Rego et al., 2022) and handwashing behaviour (Ram et al., 2014) are sensitive to the methods used to measure them. More researchers, in the sanitation field and in general, could explore the validity and reliability of outcome measures they use.

A sanitation VAS provides a simple means of capturing a respondents self-perceived level of sanitation, and we have provided evidence in support of its validity in two settings. VASs are widely used in health impact evaluation and economic evaluation, and they have potential for similar applications in the field of sanitation. Benefits of sanitation beyond infectious disease are likely to underpin willingness to pay for toilets and later upgrades, so it is important to understand broader outcomes to achieve universal access. Global measures such as VAS can capture an overall picture of how people feel about an outcome, focusing on the subjective experience of the individual. Future research priorities for VAS are assessing test-retest reliability and exploring construct validity in contexts with more diverse toilet types and larger samples.

Data availability. Datasets and Stata code are available open access online at https://osf.io/cx5av/

Contributions: IR and HHC conceived the work. ZA, BDA and KN contributed to data collection. HHC undertook data analysis for Ghana and IR for Mozambique. IR wrote the first draft, drawing on HHC’s MSc thesis. All authors contributed to critical revision of the article and approved the version to be submitted.

Ethical Approval: The Mozambique study received prior approval from the Comité Nacional de Bioética para a Saúde (ref: IRB00002657) at the Ministry of Health in Mozambique. The Ghana study received prior approval from the Committee on Human Research, Publications and Ethics, Kwame Nkrumah University of Science and Technology (ref: CHRPE/AP/317/19). The protocol for present study was reviewed by the MSc Research Ethics Committee at LSHTM, which concluded that additional ethics approval was not required (ref: 27732). Informed, written consent was obtained from all participants in both countries.

Funding. This data collection underlying this work was funded by the Bill and Melinda Gates Foundation (in Mozambique, OPP1137224) and the UK Foreign, Commonwealth & Development Office (in Ghana) via Water and Sanitation for the Urban Poor.

Conflict of interest. None declared.

Acknowledgements. We benefitted from the feedback of participants at the WASH Economics Conference 2023 and seminar participants at the Global Health Economics Centre at LSHTM. IR acknowledges the support of a post-doctoral fellowship from the Reckitt Global Hygiene Institute in the period when the paper was drafted. HHC conducted the Ghana analysis for this paper as part of a Masters dissertation at LSHTM.

Arnold, B. F., & Ercumen, A. (2016). Negative Control Outcomes: A Tool to Detect Bias in Randomized Trials. JAMA, 316(24), 2597-2598. https://doi.org/10.1001/jama.2016.17700
Bleichrodt, H., & Johannesson, M. (1997). An Experimental Test of a Theoretical Foundation for Rating-scale Valuations. Medical Decision Making, 17(2), 208-216. https://doi.org/10.1177/0272989X9701700212
Cheng, L. J., Tan, R. L., & Luo, N. (2021). Measurement Properties of the EQ VAS Around the Globe: A Systematic Review and Meta-Regression Analysis. Value Health, 24(8), 1223-1233. https://doi.org/10.1016/j.jval.2021.02.003
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. In (Vol. 6, pp. 284-290). US: American Psychological Association.
Devlin, N., Parkin, D., & Janssen, B. (2020). Methods for Analysing and Reporting EQ-5D Data. https://doi.org/https://doi.org/10.1007/978-3-030-47622-9
Dyer, J. S., & Sarin, R. K. (1979). Measurable Multiattribute Value Functions. Operations Research, 27(4), 810-822. http://www.jstor.org/stable/170296
Ernstsson, O., Burstrom, K., Heintz, E., & Molsted Alvesson, H. (2020). Reporting and valuing one's own health: a think aloud study using EQ-5D-5L, EQ VAS and a time trade-off question among patients with a chronic condition. Health Qual Life Outcomes, 18(1), 388. https://doi.org/10.1186/s12955-020-01641-4
Fayers, P., & Machin, D. (2015). Quality of Life: The Assessment, Analysis and Interpretation of Patient‐Reported Outcomes (3rd ed.). Wiley.
Freeman, M. C., Delea, M. G., Snyder, J. S., Garn, J. V., Belew, M., Caruso, B. A., Clasen, T. F., Sclar, G. D., Tesfaye, Y., Woreta, M., Zewudie, K., & Gobezayehu, A. G. (2022). The impact of a demand-side sanitation and hygiene promotion intervention on sustained behavior change and health in Amhara, Ethiopia: A cluster-randomized trial. PLOS Global Public Health, 2(1), e0000056-e0000056. https://doi.org/10.1371/journal.pgph.0000056
Gaisie, E., Poku-Boansi, M., & Adarkwa, K. K. (2018). An analysis of the costs and quality of infrastructure facilities in informal settlements in Kumasi, Ghana. International Planning Studies, 23(4), 391-407. https://doi.org/10.1080/13563475.2018.1513359
Garn, J. V., Sclar, G. D., Freeman, M. C., Penakalapati, G., Alexander, K. T., Brooks, P., Rehfuess, E. A., Boisson, S., Medlicott, K. O., & Clasen, T. F. (2017). The impact of sanitation interventions on latrine coverage and latrine use : A systematic review and meta-analysis. INTERNATIONAL JOURNAL OF HYGIENE AND ENVIRONMENTAL HEALTH, 220(2), 329-340. https://doi.org/10.1016/j.ijheh.2016.10.001
Ghana Statistical Service. (2013). 2010 Population & Housing Census: National Analytical Report.
Hawker, G. A., Mian, S., Kendzerska, T., & French, M. (2011). Measures of adult pain: Visual Analog Scale for Pain (VAS Pain), Numeric Rating Scale for Pain (NRS Pain), McGill Pain Questionnaire (MPQ), Short-Form McGill Pain Questionnaire (SF-MPQ), Chronic Pain Grade Scale (CPGS), Short Form-36 Bodily Pain Scale (SF. Arthritis Care & Research, 63(S11), S240-S252. https://doi.org/10.1002/acr.20543
Keeney, R., & Raiffa, H. (1975). Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge University Press.
Knee, J., Sumner, T., Adriano, Z., Anderson, C., Bush, F., Capone, D., Casmo, V., Holcomb, D., Kolsky, P., MacDougall, A., Molotkova, E., Braga, J. M., Russo, C., Schmidt, W. P., Stewart, J., Zambrana, W., Zuin, V., Nala, R., Cumming, O., & Brown, J. (2021). Effects of an urban sanitation intervention on childhood enteric infection and diarrhoea in Mozambique. eLife, 10, e62278-e62278. https://doi.org/https://doi.org/10.7554/eLife.62278
Lundy, J. J., & Coons, S. J. (2011). Measurement equivalence of interactive voice response and paper versions of the EQ-5D in a cancer patient sample. Value Health, 14(6), 867-871. https://doi.org/10.1016/j.jval.2011.03.001
McCarthy, M., Jr., Chang, C. H., Pickard, A. S., Giobbie-Hurder, A., Price, D. D., Jonasson, O., Gibbs, J., Fitzgibbons, R., & Neumayer, L. (2005). Visual analog scales for assessing surgical pain. J Am Coll Surg, 201(2), 245-252. https://doi.org/10.1016/j.jamcollsurg.2005.03.034
Novotný, J., Hasman, J., & Lepič, M. (2018). Contextual factors and motivations affecting rural community sanitation in low- and middle-income countries: A systematic review. INTERNATIONAL JOURNAL OF HYGIENE AND ENVIRONMENTAL HEALTH, 221(2), 121-133. https://doi.org/10.1016/j.ijheh.2017.10.018
Nussbaum, M. (2001). Symposium on Amartya Sen's philosophy: 5 Adaptive preferences and women's options. Economics and Philosophy, 17(1), 67-88. https://doi.org/10.1017/S0266267101000153
Parkin, D., & Devlin, N. (2006). Is there a case for using visual analogue scale valuations in cost-utility analysis? Health Economics, 15(7), 653-664. https://doi.org/10.1002/hec.1086
Patrick, D. L., Bush, J. W., & Chen, M. M. (1973). Methods for Measuring Levels of Well-being for a Health Status Index. Health services research, 8(3), 228-245.
Pickering, A. J., Djebbari, H., Lopez, C., Coulibaly, M., & Alzua, M. L. (2015). Effect of a community-led sanitation intervention on child diarrhoea and child growth in rural Mali: A cluster-randomised controlled trial. The Lancet Global Health, 3(11), e701-e711. https://doi.org/10.1016/S2214-109X(15)00144-8
Polit, D. F. (2014). Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual Life Res, 23(6), 1713-1720. https://doi.org/10.1007/s11136-014-0632-9
Price, D. D., Staud, R., & Robinson, M. E. (2012). How should we use the visual analogue scale (VAS) in rehabilitation outcomes? II: Visual analogue scales as ratio scales: an alternative to the view of Kersten et al. J Rehabil Med, 44(9), 800-801; discussion 803-804. https://doi.org/10.2340/16501977-1031
Ram, P. K., Sahli, M. W., Arnold, B., Colford, J. M., Chase, C., Briceño, B., Orsola-vidal, A., & Gertler, P. (2014). Validity of Rapid Measures of Handwashing Behavior: An Analysis of Data from Multiple Impact Evaluations in the Global Scaling Up Handwashing Project.
Reeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., Schwartz, C., Revicki, D. A., Moinpour, C. M., McLeod, L. D., Lyons, J. C., Lenderking, W. R., Hinds, P. S., Hays, R. D., Greenhalgh, J., Gershon, R., Feeny, D., Fayers, P. M., Cella, D., . . . Butt, Z. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research, 22(8), 1889-1905. https://doi.org/10.1007/s11136-012-0344-y
Rego, R., Watson, S., Gill, P., & Lilford, R. (2022). The impact of diarrhoea measurement methods for under 5s in low- and middle-income countries on estimated diarrhoea rates at the population level: A systematic review and meta-analysis of methodological and primary empirical studies. Trop Med Int Health, 27(4), 347-368. https://doi.org/10.1111/tmi.13739
Robinson, A., Loomes, G., & Jones-Lee, M. (2001). Visual Analog Scales, Standard Gambles, and Relative Risk Aversion. Medical Decision Making, 21(1), 17-27. https://doi.org/10.1177/0272989X0102100103
Ross, I. (2022). Using water-adjusted person years to quantify the value of being water secure for an individual's quality of life. WATER RESEARCH, 119327-119327. https://doi.org/https://doi.org/10.1016/j.watres.2022.119327
Ross, I., Cumming, O., Dreibelbis, R., Adriano, Z., Nala, R., & Greco, G. (2021). How does sanitation influence people's quality of life? Qualitative research in low-income areas of Maputo, Mozambique. Social Science and Medicine, 272, 113709-113709. https://doi.org/10.1016/j.socscimed.2021.113709
Ross, I., Greco, G., Adriano, Z., Nala, R., Brown, J., Opondo, C., & Cumming, O. (2022). Impact of a sanitation intervention on quality of life and mental wellbeing in low-income urban neighbourhoods of Maputo, Mozambique. BMJ OPEN. https://doi.org/10.1136/bmjopen-2022-062517
Ross, I., Greco, G., Opondo, C., Adriano, Z., Nala, R., Brown, J., Dreibelbis, R., & Cumming, O. (2021). Measuring and valuing broader impacts in public health: Development of a sanitation-related quality of life instrument in Maputo, Mozambique. Health Economics, 31(3), 466-480. https://doi.org/10.1002/hec.4462
Sclar, G. D., Penakalapati, G., Caruso, B. A., Rehfuess, E. A., Garn, J. V., Alexander, K. T., Freeman, M. C., Boisson, S., Medlicott, K., & Clasen, T. (2018). Exploring the relationship between sanitation and mental and social well-being: A systematic review and qualitative synthesis. SOCIAL SCIENCE & MEDICINE, 217, 121-134. https://doi.org/https://doi.org/10.1016/j.socscimed.2018.09.016
Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health Measurement Scales: A practical guide to their development and use (5th ed.). Oxford University Press.
Tidwell, J. B., Nyarko, K. B., Ross, I., Dwumfour-Asare, B., & Scott, P. (2022). Evaluation of user experiences for the Clean Team Ghana container-based sanitation service in Kumasi, Ghana. Journal of Water, Sanitation and Hygiene for Development, 2020.2010.2023.20218578-20212020.20218510.20218523.20218578. https://doi.org/10.2166/washdev.2022.013
Torrance, G. W. (1976). Social preferences for health states: An empirical evaluation of three measurement techniques. Socio-Economic Planning Sciences, 10(3), 129-136. https://doi.org/https://doi.org/10.1016/0038-0121(76)90036-7
Torrance, G. W., Feeny, D., & Furlong, W. (2001). Visual analog scales: Do they have a role in the measurement of preferences for health states? In (Vol. 21, pp. 329-334). [Cambridge, MA] :: Birkhäuser Boston.
Tumwebaze, I. K., Orach, C. G., Niwagaba, C., Luthi, C., & Mosler, H. J. (2013). Sanitation facilities in Kampala slums, Uganda: Users' satisfaction and determinant factors. INTERNATIONAL JOURNAL OF ENVIRONMENTAL HEALTH RESEARCH, 23(3), 191-204. https://doi.org/10.1080/09603123.2012.713095
Welsch, H. (2006). Environment and happiness: Valuation of air pollution using life satisfaction data. ECOLOGICAL ECONOMICS, 58(4), 801-813. https://doi.org/10.1016/j.ecolecon.2005.09.006
Whynes et al. (2008). Correspondence between EQ-5D health state classifications and EQ VAS scores. Health Qual Life Outcomes, 6, 94. https://doi.org/10.1186/1477-7525-6-94
Wolf, J., Hubbard, S., Brauer, M., Ambelu, A., Arnold, B. F., Bain, R., Bauza, V., Brown, J., Caruso, B. A., Clasen, T., Colford, J. M., Freeman, M. C., Gordon, B., Johnston, R. B., Mertens, A., Prüss-Ustün, A., Ross, I., Stanaway, J., Zhao, J. T., . . . Boisson, S. (2022). Effectiveness of interventions to improve drinking water, sanitation, and handwashing with soap on risk of diarrhoeal disease in children in low-income and middle-income settings: a systematic review and meta-analysis. The Lancet, 400(10345), 48-59. https://linkinghub.elsevier.com/retrieve/pii/S0140673622009370

The authors declare no competing interests.

Cheung2024supportinginformation.pdf
Supporting Information

Download PDF

Version 1

posted

You are reading this latest preprint version

Validity of a visual analogue scale to measure and value perceived level of sanitation – evidence from Ghana and Mozambique

Status:

Version 1

Abstract

Figures

1. Introduction

2. Methods

3. Results

4. Discussion

5. Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1