Evaluating AVADAR Mobile Health Intervention Using a Multi-prong Approach

Background The Auto-Visual AFP Detection and Reporting (AVADAR) digital health intervention programme is a programme that was introduced to Africa in 2016. The programme adopts the use of the AVADAR SMS – based smartphone application (app) in community-based AFP surveillance activities in order to enhance the detection and reporting of AFP (polio) cases and improve AFP surveillance quality. As at 2020, the AVADAR application is being used in 11 African countries. The need to conduct regular and relevant evaluations of the AVADAR programme is very essential towards improving polio eradication programme performance and effectiveness in Africa. Hence, this study aimed to review and evaluate the quality of the AFP cases reported through the AVADAR intervention and as well evaluate the documentation process of AVADAR alerts and investigations, and the assimilation of AFP cases found via AVADAR into the national databases. Methods This study reviewed and evaluated the quality of AVADAR-involved AFP case reporting and documentation process in 7 of the 11 African countries implementing the AVADAR programme (Cameroon, Chad, the DRC, Liberia, Mali, Niger, and South Sudan). Case validations of all AFP cases reported via AVADAR app, iterations of methods used for peer reviewing AVADAR reporting and documentation, informal interview of community informants (CIs) and health workers (HWs), as well as the development of interactive dashboard to showcase the results of peer reviews, were the approaches used for the review and evaluation process. Results Thirty-nine districts, cutting across the participating 7 African countries were selected for the study. A total of 581 AFP cases were reviewed in the selected districts; of which 496 AFP cases were physically seen with 384 cases conrmed as true AFP cases by the peer reviewers. Thematic ndings obtained the interview with CIs and HWs identied key areas (communication, multi-disease reporting, and periodic evaluation) that needs to be improved in the AVADAR surveillance system. Also, the interactive dashboard gave a summary of the peer review outcomes at few glances. Conclusions The ndings of the AVADAR AFP peer reviews revealed the app’s ecacy in reporting AFP cases and improving surveillance indicators at district level. However, its documentation at health facility level needs to be re-emphasized and improved via a systematic accountability framework implementation for the actors in the reporting cycle. In order to signicantly improve AFP surveillance, we recommend on-going commitment to improve knowledge and collaboration between all AVADAR surveillance reporting teams involved in identifying children presenting with AFP.


Introduction
Acute accid paralysis (AFP) surveillance is one of the four bedrock strategies for polio eradication and it includes the detection and investigation of cases of new onset of accid paralysis in children aged <15 years or any person, of any age, whom a clinician suspected to have poliomyelitis [1](2). AFP surveillance aims to ensure that there is no case of AFP in our communities, which is an essential means of monitoring the progress towards polio eradication (2) . Conferment of a polio-free status in a de ned population group (such as a district, a province, or a country) is dependent on two key pre-requisites, namely: the successful detection, by the local surveillance system, of one case of non-polio AFP per 100,000 children below the age of 15 years per year; and the achievement of a status of "no case of polio occurrence for three consecutive years", as captured by the key indicators that set the gold standard for AFP surveillance quality (3).
Routine AFP surveillance across most African countries has been fraught with challenges, often linked to limited access to hard-to-reach areas, inability of health workers to correctly and accurately identify AFP cases, poor reporting methods, as well as delays and bottlenecks associated with AFP surveillance data reporting(4) (5). The Auto-Visual AFP Detection and Reporting (AVADAR) application (app) is a digital health intervention designed and developed in 2016 by the Bill and Melinda Gates Foundation (BMGF), in conjunction with the World Health Organization (WHO), the Novel-T and the eHealth Africa (eHA) as a supplemental community surveillance system that harnesses SMS technology and automated alerting, and combines it with corresponding investigations by community informants (CIs) who are systematically and geographically selected for improved active case search and reporting of AFP cases (6) (7). The app, installed on the smartphones of health workers and community informants, widens the AFP surveillance network outside that of the priority reporting sites in order to improve the sensitivity and quality of AFP surveillance, more so in areas (particularly districts) with low performance indicators. The core criteria for choosing a district for the use of AVADAR app include (8)(9)(10)(2) With investigation rates averaging above 95% for all AFP cases reported by CIs enrolled in the AVADAR digital health intervention, through the AVADAR app, it was observed that it will be of vital importance to conduct a peer review to evaluate and test the validity of these results. The premier AVADAR peer review was done in DRC and it sought to document the best practices from the implementation in DRC, review the AVADAR true cases and proffer good data management solutions where necessary. The ndings obtained from this premier district-level peer review precipitated a bigger peer review series covering some of the districts in the African region where the AVADAR digital health intervention project has been implemented. All these districts are situated within 7 AVADAR-implementing countries, they are: Cameroon, Chad, DRC, Liberia, Mali, Niger, and South Sudan.
This paper documents the processes of the peer review and evaluation, and validates results of AVADAR intervention in the focused AVADAR-implementing districts in 7 countries in the African region. It also seeks to evaluate the reliability of the data in these districts while outlining the data quality issues and bottlenecks in the system with an improvement plan (14) (15)

Methods
The peer review and evaluation of the AVADAR digital health intervention was phased, planned, and conducted over a period of one year: started in September 2018 (commenced at DRC) and concluded in September 2019 (at South Sudan).
A standard operating procedure on the selection of AFP cases, recorded through AVADAR app, to be reviewed in the AVADAR-implementing districts was developed, drafted and adopted by all the stakeholders involved. The number of children to be assessed for veracity of true AFP cases reported was extrapolated via calculations of best sampled value for the total number of true AFP cases and then the geographic spatial representation of the children was analysed for a better geographic coverage of the evaluation.
An electronic mobile form was designed to automatically collect geographic data during the peer review alongside the veri cation details of the AFP cases reported via the AVADAR app. This mobile form was built on the Open Data Kit (ODK), with all the considerations for low resource settings (16), and hosted on the regional WHO Mobile data aggregation server for ease of access in all the countries conducting the peer review.
The peer review comprised of a wide range of participants from different agencies already familiar with AFP Surveillance for transparency and open discussions on the results of the evaluation (17,18). The agencies included WHO, MOH, UNICEF, BMGF and eHA. Each eld peer review team comprised at least 3 members from one of the core 5 evaluation agencies. Importantly, the CI who issued the AFP alert (through AVADAR app) would accompany the team to review the respective AFP case. Debrie ng was carried out after the review and data validation exercise were conducted with the State Ministers of Health or a designated o cial at that level of administration.
Mobile phones preloaded with the electronic peer review form and a printed copy of the database of children preselected for evaluation with the assigned unique Epidemiological Number (EPID Number) (19) for AFP cases, the physical address of the child and reference serial identi er (rsIDs) [1].

Case Validation
The investigations of AFP cases reported through the AVADAR app were originally carried out by a member of the District Management Team, sometimes accompanied by the AVADAR Programme Coordinator or his deputy. The EPID number and the name of the CI who noti ed the case were recorded on the investigation sheet. All investigation sheets and case investigation sheets are archived in the health facility.
The peer review focused on validation of these original AFP investigation sheets through an electronic validation process that involved a clinical assessment of the AFP case to determine if the report of the initial AFP case investigation by the local team investigation team was true.
The parents or caregivers were also visited systematically and questions were asked while the child was physically examined to validate the original diagnosis by the diseases noti cation o cer (20)(21) Iterations of the Peer Review Methods In conducting the reviews within the 7 participating countries, 4 iterations of the peer review processes were conducted to cover the limitations of the methods used in each stage of the peer review. A table of the peer review iterations and the countries it was deployed in is shown below: Community Informants and Health Workers Interviews: Facilitators of the peer review informally interviewed a randomly selected sample of health workers enrolled in the AVADAR investigation cycle as well as a randomly selected sample of CIs who originally reported the AFP cases on some of the unstructured elements of the AVADAR intervention to sieve out programmatic hindrances and/or enablers of the system with 3 main elements for consideration: 1. Ease of communication.

Periodic evaluation
A total of 55 Health workers and 64 CIs were randomly interviewed using a question guide that captured the above-listed thematic areas. Relevant ndings obtained from the interview were grouped into these three (as above) thematic areas and summarized.

Interactive Dashboard
An interactive results dashboard was developed and hosted using ArcGIS online to showcase the results of the peer review across the districts and countries. The dashboard collated all the data into a summary accessible table and charts and also, it was provided for stakeholders to interact with the AVADAR peer review data straight from the eld after minimal automated data cleaning and validation.
[1] rsIDs: reference serial identi er is an identi cation tag assigned to a suspected AFP that maps to an alert from a community informant

Case Validation
Out of the 581 children, from the 7 countries, that were included in the AVADAR peer review, only 496 (85%) were physically seen during the process. Among the children seen, 384 (77%) children were evaluated to be true AFP cases; and of those, 35 (9%) were found to have residual paralyses [1]. The other 112 (23%) children were non-true AFP cases -they were misdiagnosed cases of spastic paralysis, malnutrition and trauma (22)(23). Of the 85 (15%) children who were not physically seen during the peer review visits, 39 (46%) had died, 25 (29%) had travelled out of their original locations, 11 (13%) were not found with the reasons being undocumented, and 10 (12%) were guest children that sought help in the AVADAR-implementing communities. Most of the children evaluated were found to be malnourished and to have weak muscles of the lower limb (20,21). The basic muscle tone was intact, however, walking remained di cult and these children who were originally considered as true AFP cases were subsequently validated as non-true AFP cases after the review.
Most of the children evaluated were found to be malnourished and to have weak muscles of the lower limb (20,21). The basic muscle tone was intact, however, walking remained di cult and these children who were originally considered as true AFP cases were subsequently validated as non-true AFP cases after the review. Below is the summary of the thematic ndings obtained from the informal interviews conducted by the peer review facilitators among the CIs and health workers: 1. Ease of Communication: the interviews revealed that all end-users of the AVADAR intervention agreed that the Closed User Group (CUG) was the most important must-have component that was an irreplaceable element as it assisted in producing the remarkable results of improving AFP surveillance indicators in the AVADAR districts.

Multi-Diseases
Reporting: the interviews revealed that AVADAR app was found to also report on other diseases such as cerebrospinal meningitis (CSM), yellow fever, and Lassa fever. However, there was a lack of structured reporting like that of AFP which it was originally designed and develop for.
3. Periodic Evaluation: the interviews revealed that monthly meeting were found to provide an opportunity for zero reporting(24) reviews and discussions of limitations identi ed by the participants in the AFP reporting processes. Also, all 39 districts in the 7 countries assessed during the peer review a rmed to attending the monthly meetings and saw it as an opportunity to discuss openly and interface on issues around the implementation of community based surveillance.

Interactive Dashboard
The results of the evaluation were mapped dynamically to an interactive dashboard to showcase the ndings by district, province and country (25). The dashboard displayed the minimum required data to assess the quality of the peer reviews and the added value of AVADAR in each district. Some common steps identi ed in the evaluation process were: (i) de ning the aspects of the surveillance system under evaluation, (ii) designing the evaluation process with electronic tools (iii) implementing and iterating the evaluation, and (iv) drawing conclusions and recommendations.
An example of the charts and maps created as well as the interactive dashboard are shown in the gures below:

Discussion
This evaluative review was conducted to establish the quality of AFP cases reported through the AVADAR app network. During the review, we noted that 85% of all AVADAR alerts were investigated within 7 days. The AVADAR cycle set-up ensured that as soon as the CI detected a suspected AFP case, they sent an alert and called the nurse/lead investigators to conduct the investigation. There were times when the rsID number arrived later due to SMS congestion on the network; however, this did not deter the investigations as the CIs used the Closed User Group (CUG) setting of the app to immediately call in a suspected case. An example of the successes scored by the AVADAR network is the DRC, where the network averaged a 98% investigation and validation rate of cases which are conducted within 48 hours of reporting. This is a demonstration of the country's commendable diseases reporting system which requires that case investigation and validation are carried out as early as the same day of detection (11). Thus accountability of staff investigating alerts and documenting the AFP cases is a very important consideration for the success of the AVADAR system The name of the CI, coordinates, as well as the EPID number of all AFP cases were clearly mentioned in the health system AFP document (Investigation sheet, validation sheet). Unfortunately, not all the archives of AFP cases were well maintained in every health centre. AVADAR documentation in the facilities can be improved signi cantly by ensuring that all hard copies of AFP investigation forms have AVADAR clearly marked on the forms. The CIs were well motivated, but they faced a number of problems particularly at the beginning of the implementation, including uncharged phones as a result of the bad quality solar power banks provided. Eventually, they resorted to using electricity to charge their phone particularly on Saturday and Sunday so as to avoid missing the Zero report deadline required on Monday. Additionally, the e-Health teams that were present during the review contributed to the performance of the CI activities. The support of eHA through their commitment to assist with the investigation processes translated into the warm welcome the peer review teams received in homes of AVADAR AFP cases during the review.
The AVADAR team structure was well designed and the support they provided to community health systems such as the health zones management teams (HZMT) was well planned and executed. Their collaboration with the HZMT was a key performance factor. During the review, there was great synergy between the AVADAR team and the HZMT in all the seven countries. We did, however, nd that the quality of the investigation or validation was affected by neurological assessment knowledge gaps within the teams; this limited their ability to better assess the true cases from among the other palsies or weaknesses. This lack of neurological examination skills required to con rm paralysis and then accid paralysis at the investigation level led to some misdiagnoses for true AFP, with malnutrition being clearly the cause of weakness of muscles of these AFP cases. For the CI, according to de nition, those cases were deemed to be true AFP cases but for the medical team and health workers, the assessment was able to differentiate between weakness and paralysis. As such, we observed that the propensity of the AVADAR investigation team was to con rm all suspected AFP cases reported as true without question, in accordance with the community de nition. The review showed that the local team validated the alerts as AFP case when they should have given a different assessment. Thus, the validation process at local level was found to require re-examination to ensure fool-proof AFP true cases. Therefore, reliance was placed on the clinical assessment conducted for the case to provide additional information to make a correct determination.
A notable improvement that can be achieved at district level is to inculcate training and equipping of health workers to improve their knowledge of disease surveillance and identifying AFP cases properly. It is critical that AVADAR improvement process cater to the shortcoming of validation of true AFP and strategic spread of informants' active case search.
Furthermore, we found that although the HZMTs performed the traditional system of case reviews using the symptoms column in the consultation registers in health areas, there were some weak points, particularly that these reviews were not conducted periodically as required by the site level of surveillance.
It is possible that the high performance of AVADAR investigation rates as seen in some of the countries like DRC could be masking bad quality of the AFP case validation process. Thus, the results on the server from the electronic processes of AVADAR were re-validated on a case by case basis with a minimum of 50% of the sample size chosen for the peer reviews. For districts with high population density and easy thorough fare for movement, the peer review teams were able to see all(100%)of the children in the sample size.
Of additional note is that in most health systems, the CIs are envisaged to be volunteers and work pro bono for their activities. They do, however, receive a small compensation as motivation for their additional time spent in the AVADAR activity and therefore more remunerated than their counterparts doing normal active surveillance. This could precipitate in sustainability issues if the local health authorities do not take up ownership of AVADAR surveillance as the Polio End game ramps down.
The interactive dashboard served as rst line decision making tool for remote desk support and advisories from the country and regional level colleagues to view the district level peer review performances in near real-time. The country teams found it very useful especially in viewing spatial representation of where the children were visited and the status of aggregated district level peer reviews.

Conclusions
There are major lessons to be learnt from the AVADAR peer reviews in validating the gains of the AVADAR intervention for improving surveillance. The rst is the immense contribution of the CUG utilization clearly elucidated by the alerting and investigation cycle turnaround time. There is an urgent need for neurological examination training for key health workers at district level. With this improved knowledge by the health workers, the inherent transfer of knowledge from the health workers to the CIs will reduce the issue of false positives and false negative cases found via the AVADAR system. Invariably this will reduce the burden of investigations currently faced by the investigators.
The role of active case search by informants when boosted by AVADAR alongside sustained investigation cycles and government involvement can lead to sustainably improved AFP detection rates. Additionally, transitioning the app to a fully multi-diseases reporting platform may prove bene cial for other diseases in the surveillance systems.

Declarations
Ethics approval and consent to participate: The Ministries of Health in each country gave approval for this study. We used primary data collected by the research team. Consent was obtained from all study participants to ensure that all respondents had a common understanding of the consent requirements.

Consent for publication : N/A
Availability of data and materials: All data generated and/or analyzed during this study are included in this published article. The data are available from the corresponding author upon request.