Results are organized by the implementation factors of time, partnerships, financial costs, and technology and infrastructure, with key findings from each country case presented.
Time
Current EIR guidance documents provide an overview of steps to plan for in the EIR process, but do not include details on timelines associated with each step or the overall process as it can vary widely.
Vietnam
In Vietnam, it took seven years to fully design, develop, and deploy the EIR from its initial design as part of the Optimize Project beginning in 2010 (24) until it reached nationwide scale in 2017. Over this period, the EIR matured through a series of phased digital health pilots, each of which provided an opportunity for iterative learning and improvement (Table 2).
The initial design phase for ImmReg took 3-4 months to conduct a landscape assessment, collect and document user system requirements, and collaborate with the software developer to finalize the technical specifications. The development phase for ImmReg took another 4 months to develop the software, including user interface testing and case-testing to check the functionality and data accuracy. In comparison, Vaxtrak took approximately 6 months to design and another 6 months to develop, this longer process was due to the more complicated system requirements for vaccine management and reporting. Deployment of the initial versions of ImmReg and Vaxtrak began in 2012 in a limited number of provinces.
From 2013-2014, ImmReg and VaxTrak were upgraded and combined into a single web-based application. Many new functional requirements were added, but the time to develop the requirements documents was minimal (approximately 2 months) because the system workflows, interface, functionalities, architecture, and end-user needs were well understood from the previous implementations. The development of the combined system took approximately one year, and the updated system was deployed beginning in 2014.
In 2016, the combined system was integrated into the NIIS which saved the NIIS software development team time by integrating the knowledge and technology from the previous pilots. The development of NIIS took approximately 8 months including in-house testing before deploying first in 5 pilot districts and then nationwide. As of 2017, the NIIS has been introduced in all facilities across Vietnam, although support to encourage system uptake, improve data quality, and transition to paperless is ongoing.
Tanzania
In Tanzania, the EIR requirements were shared in a request for proposal in late 2014 and the first system, TIIS, was chosen. TIIS development was initially planned for 3 months but due to the complexity of the system being developed, it was extended to six months (Table 2). As mentioned above, technical challenges with TIIS emerged during the pilot in Arusha Region which led to the decision to update the requirements and release a revised RFP in February 2016 (37). The OpenIZ system, TImR, was selected in and developed over approximately 9 months in 2016-2017. TImR deployment began in 2017 and the government plans to scale TImR nationwide to all facilities by the end of 2021.
Zambia
In Zambia, the EIR requirements were shared in a request for proposal in July 2015 and the DHIS2 Patient Tracker software was selected and began development in 2015. Like the Tanzania experience, there were challenges adapting the software to meet the requirements, and development stopped after 10 months in 2016 (38). After refining the requirements, Zambia selected the OpenSRP-based ZEIR in early 2017. OpenSRP had already been successfully used for immunization in Pakistan which contributed to the efficient development in Zambia, requiring 3 months of development time for an initial release and 6 months for a near-complete version for expanded rollout (39). Deployment began in May 2017 and ZEIR was deployed to 320 facilities in Southern Province by March 2018. An additional 281 facilities in Western province are yet to go live and there are plans to scale ZEIR nationwide.
Partnerships
EIR guidance documents recommend developing a multidisciplinary team that can oversee all project functions and defining the roles and responsibilities of each participant from the outset (16–18). The guidance also notes that some partners with specific skillsets (e.g., in software development or training) may only be required for certain project phases.
Vietnam
Beginning in 2010, NEPI, PATH, and WHO partnered to develop ImmReg and VaxTrak. A challenge that surfaced during the initial work in 2010-2012 was that IT professionals lacked immunization expertise, and public health professionals lacked understanding of technical software development terms and processes. A lesson learned was to have “bridging” staff who could improve alignment, for example, including an IT professional with technical expertise on the core project team to improve collaboration with software developers (24).
In 2016, when the NIIS was being re-designed to scale from one province to 63 provinces nationwide, the NIIS Technical Working Group (TWG) was formed. The TWG includes representatives from appropriate agencies with complementary expertise:
- The MoH General Department of Preventive Medicine (GDPM) provides management authority, stakeholder coordination, administration support, and oversight of provincial implementation and policy change.
- The NEPI provides immunization expertise, acting as the technical lead on immunization workflows, corresponding user requirements, and reporting systems.
- Viettel, one of the largest mobile network operators in Vietnam, provides technology expertise to develop and maintain the system.
- PATH provides technical support and facilitates communication between TWG members, serving as the connection between immunization, general health, and technology stakeholders.
All partners bring complementary expertise and have been crucial to the scale-up of the NIIS from design to deployment. Although the TWG partnership has functioned well, there was initially confusion in roles, deliverables, and financial responsibilities which created delays in coordination and implementation (40).
In addition, partners at subnational levels were involved in the design phase to inform system requirements, in the development phase to support testing the system and providing inputs on user experience, and in the deployment phase for training and ongoing mentorship. During design and development, representatives of different end-user roles were included, such as managers, EPI officers, and storekeepers.
Tanzania
Through the BID Initiative, PATH partnered with the MOHCDGEC. An MOU was signed in September 2014 to formalize the relationship. During the design and development phases, PATH worked closely with the national IVD program, the MoH Information Communication Technology (ICT) Unit, as well as regional and district leadership, particularly in the pilot site, Arusha Region. PATH provided technical oversight and business analysis support, while the MoH was responsible for testing and ensuring the EIR met the use case requirements.
During the EIR development phase, Mohawk College (Canada) was selected to develop TImR. However, not having the lead developers in-country made communication challenging due to time zone differences and a limited understanding of the local use cases. When there were critical system issues, the fixes were often delayed due to the time zone differences and limited capacity outside of the software teams. Mohawk College partnered with an in-country software firm, Softmed, to improve responsiveness and build capacity for ongoing system maintenance and sustainability.
At the subnational level, a User Advisory Group (UAG) provided end-user perspectives on the system requirements and tested the usability of the system during the design and development phases. UAG members included representatives from regional, district, facility, and community levels who support immunization service delivery, supply chain, data collection, or community engagement (41).
Other partners included those in the Inter-agency Coordinating Committee (WHO, UNICEF, and other partners) who monitored EIR progress.
Zambia
Through the BID Initiative, PATH partnered with the Ministry of Community Development, Mother and Child Health (MDCMCH) and signed an MOU on February 5, 2015 to formalize the relationship. While the EPI unit was under the MDCMCH, the management of information and communication technology (ICT) and M&E were under the MoH. This structure was challenging to coordinate decision-making between two ministries which led to delays in the EIR design, development, and implementation process. A technical working group was formed in 2015 that included staff from the EPI, M&E, and ICT units from both ministries which facilitated streamlined decision-making. In 2016, the Mother and Child Health unit shifted to fall under the MoH and a new MOU was signed.
In practice, PATH worked closely with provincial and district leaders in Southern Province, the area selected to pilot the EIR. Like Tanzania, a UAG was established that included district, facility, and community level end users to provide inputs on the system design and development (42). In the EIR development phase, Kenya-based software developer Ona was contracted to develop ZEIR and a local software company, Blue Code, was also contracted to support ongoing maintenance of the system. Like in Tanzania, having a local software partner helped address communication, contextualization, and sustainability challenges in working with a software developer based outside Zambia.
Beginning in 2017, the Catholic Medical Mission Board partnered with the MoH and PATH to deploy ZEIR to nine health facilities in Mwandi District of Western Province (43).
Financial costs
EIR guidance documents emphasize the need to develop a funding model that can sustain the project through the entire process, including all organizational costs in the short, medium, and long term (16–18).
Vietnam
During the initial ImmReg design phase, the project team developed a financial model to predict the costs of ImmReg implementation in Vietnam. Utilizing this model, the costs were estimated for scaling to forty provinces with 450 districts and 9,000 communes over a five-year period. The model generated costs for each year at commune, district, and provincial levels. This was an important input to the planning and design of ImmReg.
Since Viettel joined the NIIS TWG, they have provided in-kind services to support the development and deployment of the NIIS. Although Viettel is committed to supporting the NIIS, they have raised the challenge of ongoing investment in infrastructure and human resources (40). The Vietnam MOH and Viettel are discussing an appropriate mechanism and service fee that will allow Viettel to operate, maintain, and upgrade the system.
During the deployment phase, user feedback resulted in unanticipated costs to update the NIIS and to provide supportive supervision visits to address technical issues. At the beginning of nationwide scale-up, PATH and WHO financed the training-of-trainer trainings for national, regional, and provincial level staff, as well as training for district staff in 20 provinces. Overall, the pilot phase was longer than planned which resulted in higher costs than originally projected.
In terms of ongoing operational costs, local governments are responsible for covering the cost of maintaining or replacing equipment (PCs, barcode scanners, and printers), conducting refresher trainings, providing supportive supervision, and funding internet or 3G connectivity. One ongoing cost that was not factored into the original financial model, is the need to update and expand the NIIS server to accommodate the enormity of data as additional children are registered into the system.
Tanzania and Zambia
When the BID Initiative launched in 2013, there was limited evidence on the cost of designing, developing, and deploying EIRs – particularly in low-resource settings. Recognizing this evidence gap, the BID team used a total cost of ownership model to estimate the costs to design, develop, deploy, and maintain EIRs in Tanzania and Zambia. This costing work focused on the costs incurred during the duration of the BID Initiative between 2013 and 2018. The results showed that hardware and rollout costs were the two largest shares of the overall costs (44). However, for each deployment to subsequent regions in Tanzania, costs declined by implementing the learnings gained from previous deployments.
The total annualized cost of developing, deploying and maintaining the EIRs and their related suite of interventions through the BID Initiative was estimated to be between US$3.30 to $3.81 per child for the regions in Tanzania and US$8.46 in Zambia (44). The Zambia estimate is higher in part due to a smaller birth cohort compared to Tanzania. Additionally, a micro-costing study estimated the annual cost savings per facility resulting from the introduction of the BID interventions (including the EIR) as US$10,236 in Tanzania and $628 in Zambia, largely driven by health worker time savings in service delivery and reporting (7).
During the deployment phase, there were significant financial cost implications based on the implementation approach used. At the outset in Tanzania, the PATH team led the deployment through a series of visits to health facilities for on-site training. However, this approach was too expensive and technically infeasible to scale and necessitated a new approach whereby PATH staff trained district data use mentors who led the facility-level deployment. The new approach improved country ownership and reduced implementation costs by 10-65% per district (45).
To encourage sustainability and government ownership, the MOH in each country began to take over procurement costs toward the end of the BID Initiative project, buying tablets, providing project vehicles, and using government staff for trainings. In both countries the districts are budgeting for ongoing operational costs, like data bundles and tablet replacements, in an increasing fashion. Tanzania has secured Gavi Health System Strengthening funding to continue to scale the EIR and Zambia has secured Gavi funding for an EPI Optimisation Platform (EPI OPT) that includes strengthening ZEIR. In Zambia, a second EPI OPT grant is underway to support scaling ZEIR to additional 281 facilities in 16 districts of Western Province.
Technology and infrastructure
EIR guidance documents highlight the importance of taking into account the infrastructure needs to inform the selection of and support for a new technology (16).
Vietnam
In the design phase, the initial ImmReg landscape assessment considered aspects of the technological infrastructure including IT knowledge of health workers, lessons learnt from previous software implementations, and available equipment and connectivity. The results in 2011 showed that commune health centers did not have computers nor internet connectivity, there were limited resources available to purchase computers, and health workers’ computer skills were limited but they were comfortable using mobile phones. Based on this, ImmReg and Vaxtrak were designed as stand-alone mobile applications for commune health workers with a web-based interface for the district level where computers were available. However, despite health workers’ comfort and access to mobile phones, they were not familiar with downloading and installing new software which created challenges. By 2013-14 when ImmReg and VaxTrak were combined and upgraded, most commune health centers had computers and internet connection, so the combined system was designed as a web-based application.
The deployment phase has required ongoing technical support and system maintenance. The system has required continuous editing to add new vaccines, new variables, or new data queries based on end user feedback. Over time, Vietnam shifted from providing in-person technical support for end users to remote technical support, this has saved time, improved responsiveness, and decreased the number of field trips for the technical team. Currently, remote support includes an artificial intelligence chatbot, a telephone hotline, technical support groups on a popular social media application (Zalo), and trained mentors at each level of the health system. In addition to support from the health system, Viettel has an end-user support network and 24/7 hotline which ensure timely technical support for end users.
Tanzania
At the outset of the BID Initiative in 2013, eHealth was a national objective and well supported by the President and Minister of health. The MoH was in the process of establishing an eHealth Steering Committee and eHealth strategy and there was mainly 2G internet connectivity in most urban places. However, digital literacy was very low particularly among frontline health care workers, most facilities did not have computers, and some facilities were not connected to the national electricity grid nor internet services. The UAG helped to advise on the existing technological infrastructure during the design phase.
The existing infrastructure informed decisions during the design phase to require online and offline functionality and to purchase tablets for most facilities. For facilities without reliable access to electricity, a paper-based version of the EIR was designed that could be scanned at the district level to digitize the data. However, during the pilot phase it was determined the paper-based forms were not feasible, and all facilities received tablets to use the EIR. Those facilities with intermittent electricity also received solar chargers for the tablets.
During the deployment phase, district staff found it challenging to train health care workers due to their varied familiarity and confidence with technology. Another challenge was in purchasing and distributing sufficient data bundles to support tablet connectivity (46). In some cases, there were also challenges with syncing the tablets to the EIR.
Currently, the MoH team provides ongoing technical support for software development and maintenance, as well as data analysis and use. Immunization staff at each level of the health system provide technical support to end users at lower levels and there are district WhatsApp groups that facility health care workers use to raise issues or troubleshoot.
Zambia
The UAG in Zambia advised on the technological infrastructure during the EIR design phase. In the pilot region of Southern Province, 144 (48%) of 302 health facilities were known to have no consistent power source (46). Based on the existing infrastructure during the design phase, the BID Initiative team initially thought that some facilities would need to continue to use improved paper-based registers, however it was later determined that to ensure cost-effectiveness of the EIR all facilities should use the electronic system. Solar chargers were procured for facilities with intermittent electricity.
Like Tanzania, the EIR deployment in Zambia faced challenges resulting from limited technology skills among health workers, difficulty securing data bundles, and inconsistent data syncing (46).
Currently, BlueCode is working with the MoH to build its capacity for both technical and governance ownership of the project. This entails working closely with the MoH technical and management team and supporting the MoH with supervision and training. There are also ongoing iterations to strengthen ZEIR based on user feedback and as mentioned, ZEIR has been integrated with other systems through the EPI OPT.