To our knowledge, this is the first study assessing the inter-rater reliability of MAPS-Global in an African urban context, and our findings highlight the importance of local knowledge in applying research tools effectively. Our findings suggest that auditors with local familiarity yielded more reliable audits compared to their international peers. Despite the global accessibility of virtual platforms like Google Street View and Google Earth for environmental assessment, our results underscore the value of contextual familiarity in enhancing the meaningful application and rigor of research tools.
Incorporating contextual familiarity in global health research is crucial and at the same time an ethical responsibility when the tools we use have not been validated in the contexts where we work (Canelas et al., 2024). This practice risks oversimplifying complex realities and may lead to wrong or misleading conclusions. Rzotkiewicz et al., (2018) highlighted this gap, noting the absence of studies using Google Street View in Africa and limited research in Latin America and Asia, challenging the assumption of universal applicability for virtual audits.
This study contributes from an African setting to the limited and inconclusive research on rater familiarity in evaluating the built environment. Two studies, one utilizing the MAPS-Global in Belgium and the other applying S-VAT tool in Norway, demonstrated that auditors with greater contextual familiarity or those conducting audits in-person reported higher inter-rater reliability (Andersen et al., 2021; Vanwolleghem et al., 2016). On the other hand, Fox et al., (2021) using MAPS-Global in five HICs countries and Zhu et al., (2017) using MAPS-Global in the US suggested that familiarity does not significantly affect virtual audit outcomes. Most studies that have used virtual tools to characterize the built environment have been carried out in HICs with different environmental characteristics compared to LMICs (Andersen et al., 2021; Curtis et al., 2013; Fox et al., 2021; Kelly et al., 2013; Vanwolleghem et al., 2016; Zhu et al., 2017). Their studies present findings from well-planned cities making it difficult to draw similar conclusions to a highly urbanized and dynamic environment of the Soweto township.
Not only are LMICs underrepresented in virtual auditing, but there are also acute spatial inequalities in the amount of GSV coverage within cities depending on deprivation levels (Fry et al., 2020). Several authors have stated that virtual audits are a reliable alternative to in-person street audits but with a caveat that there is the need for high coverage and updated images (Fox et al., 2021; Vanwolleghem et al., 2016). Our study tackled the variability in coverage by selecting routes with at least 75% visibility. However, assessing the recency of GSV images posed a challenge, as image dates can vary widely even within the same location, depending on the viewing angle. Incorporating insights from a local team regarding acceptable image year ranges can significantly enhance the relevance of urban assessments, especially in LMICs where rapid urbanization is prevalent (Ritchie et al., 2024; UN-Habitat, 2022). This dialogue with the local auditors is crucial due to the continuous and fast-paced urban changes, underscoring the necessity for up-to-date imagery in environmental audits of the built environment.
Our auditors faced several challenges, including issues with image quality, outdated images, blurriness, and obstructions, similar to studies elsewhere (Andersen et al., 2021; Fox et al., 2021; Rzotkiewicz et al., 2018). An unexpected challenge emerged from feedback sessions: some resorted to using search engines to identify unclear elements in images. This practice potentially introduced inconsistencies in the auditing process, emphasizing the need for clearer guidelines in the training to ensure uniformity in virtual environmental assessments (Fox et al., 2021; Griew et al., 2013; Gullón et al., 2015). Although it is not unusual to find a high percentage of absence in some features of the microscale (Fox et al., 2021; Phillips et al., 2017), we found our study to lack many of the features of MAPS-Global. This highlights the lack of many essential amenities in these low-resourced settings but also raises the concern whether MAPS-Global was indeed the correct tool for our study site. The choice of MAPS-Global, was made collectively by the GDAR Network members (GDAR, 2023). To enhance representativity, future studies using global audit tools should consider the differences within and between neighborhoods, regions, and countries. It is important to note that developmental patterns, urbanization levels, land uses, and socioeconomic statuses of residents have different definitions, interpretations, and representations across the world.
Similar to other virtual audits of the built environment, the subjective features of the built environment such as the streets or building aesthetics had the lowest IRR across all the MAPS-Global measurements (Andersen et al., 2021; Fox et al., 2021; Gullón et al., 2015; Zhu et al., 2017). Our findings indicated inter-rater reliability was highest in the land use section of the routes, similar to results from Zhu et al., (2017). The high frequency of null responses in the crossing section (indicating a lack of infrastructure to facilitate road crossing) is notable in a country where pedestrians constitute almost 40% of road traffic fatalities (International Transport Forum, 2019). Additionally, a study in a low-income community in South Africa, showed that half of the children walking to school alone report experiences with pedestrian collisions (Koekemoer et al., 2017).
We acknowledge the limitations of our study posed by a small sample size, due to a combination of challenges, such as image coverage and limited time resources. While certain measurements, such as the pedestrian buffers, showed high percentage of agreement across all auditors, the statistical measures of reliability, such as ICC or Kappa, indicated lower values. This discrepancy stems from limited variability in exposure, where we obtained low ICC or Kappa despite a high percent agreement (McHugh, 2012; Zhu et al., 2017). However, the adoption of virtual auditing markedly decreased the costs associated with conducting audits, offering a more economical alternative to traditional in-field methods. The auditors categorized as field familiarity for this study conducted in-field audits in the same area nine months prior, and the virtual audits were completed faster (unpublished data). Additionally, while in-field audits necessitated pairs of auditors for safety reasons, virtual audits allowed individuals to work solo, providing flexibility and the comfort of conducting audits from any location.