Impact Evaluation of the Kenya Frontline Field Epidemiology Training Program

Background In 2014, Kenya’s field epidemiology and laboratory training program (FELTP) initiated a 3-month-long field-based frontline training (FETP-F) for local public health workers. Methods Between February and April 2017, FELTP conducted a mixed-methods evaluation to examine outcomes achieved among 2014 and 2015 graduates of the trainings. Data quality assessment (DQA) and data consistency assessment (DCA) scores, on-time-reporting (OTR) percentages, and ratings of the training experience were the quantitative measures tracked from baseline and then at 6-month intervals up to 18 months post-completion of the training. The qualitative component consisted of semi-structured face-to-face interviews and observations. Quantitative data were analyzed using one-way analysis of variance (ANOVA). Qualitative data were transcribed and analyzed to identify key themes and dimensions.Results One hundred and three graduates were included. For the qualitative component, we reached saturation after 19 onsite interviews and observation exercises. ANOVA showed that the trainings had small but significant impacts on mean DQA and OTR scores. Results showed an insignificant increase in mean DCA scores. Qualitative analyses showed that 68% of respondents acquired new skills, 83% applied those skills to their day-to-day work, and 91% improved work methods. Conclusion The findings show that FETP-F is effective in improving work methods, facilitating behavior change, and improving key public health competencies. high inconsistencies, whereas a high score indicates low inconsistency levels of the data between the 3 repositories of surveillance data. We used one-way ANOVA to determine if there were differences in the mean DCA scores over time.


Introduction
Strengthened health systems played a key role in the rise in global life expectancy that occurred throughout the 20th century [Roka et al., 2017]. The health workforce is the backbone of each health system, which facilitates the smooth implementation of health action for sustainable socio-economic development [Jones et al., 2017]. Developing the public health workforce in low-to-middle-income countries is a global priority. Workforce competencies and public health agency quality have Together with its partners, FELTP offers a structured local health worker training process ( Table 1).
The project focused on integration of frontline training as part of the FELTP pyramid (Fig. 1). The first phase of frontline training was implemented between September 2014 and December 2016 throughout all 47 counties in Kenya.
The goal of FETP-F was to stimulate improvements in local frontline health workers' ability to detect, report, and respond to unusual health events.
FETP-F was designed to improve overall knowledge, skills, and improvements in day-to-day work methods in the work place. Based on this hypothesis, we focused our evaluation on the expected outcomes and associated impact indicators outlined in Table 2.  Table 1b. FETP-F expected outcomes and key indicators Expected outcomes Indicators Improved skills in field epidemiology, surveillance, and data analytics Net score improvement on pre-and postself-evaluation scale Improved quality of health facility data Net improvement in DQA scores measured at baseline, 6 months, 12 months, and 18 months post-graduation Improved consistency of surveillance data within their programmatic area or field project topic area Net improvement in DCA scores measured at baseline, 6 months, 12 months, and 18 months post-graduation More and better interaction with data generated by the participant at their work place and/or within their programmatic practice arear Improved proportion of quarterly on-time submission of MoH form 753 to the county health department at baseline, 6 months, 12 months, and 18 months postgraduation 5 Improved skills in communicating public health data Number of abstracts written for health conferences Number of abstracts accepted at the submitted conferences The number of manuscripts written for publication in peer review journals The number of manuscripts accepted for publication in peer review journals This evaluation estimates the impact of FETP-F, which targets governmental local (county, sub-county, and health facility) health workers (medical and clinical officers, nurses, laboratory scientists, health information and public health officers, and veterinarians) in Kenya. Table 2 FETP-F expected outcomes and key indicators Expected outcomes Indicators Improved skills in field epidemiology, surveillance, and data analytics Net score improvement on pre-and post-selfevaluation scale Improved quality of health facility data Net improvement in DQA scores measured at baseline, 6 months, 12 months, and 18 months post-graduation Improved consistency of surveillance data within their programmatic area or field project topic area Net improvement in DCA scores measured at baseline, 6 months, 12 months, and 18 months post-graduation More and better interaction with data generated by the participant at their work place and/or within their programmatic practice arear Improved proportion of quarterly on-time submission of MoH form 753 to the county health department at baseline, 6 months, 12 months, and 18 months post-graduation Improved skills in communicating public health data • Number of abstracts written for health conferences • Number of abstracts accepted at the submitted conferences • The number of manuscripts written for publication in peer review journals • The number of manuscripts accepted for publication in peer review journals Methods Between February and April 2017, FELTP used quantitative, semi-quantitative, and qualitative methods to evaluate all FETP-F activities. Groups 1-6 formed the population for the process evaluation because they graduated ≥ 18 months before the impact evaluation began. We followed-up with graduates of these 6 groups to gather the data for the quantitative portion of the summative evaluation. Field project data provided baseline information for the DQA, DCA, and OTR measures. By Afterward, during the process evaluation at 6 months, we gauged their measures at that time. One year post-graduation, we gauged the same measures again. The final measure was taken at least 18 months post-graduation. All measures were self-reported via online survey. For all quantitative measures, we conducted one-way ANOVAs using MS-Excel's data toolpak. The quantitative measures are described below.
DQA scores. The participants had to complete a DQA for their field project, and we used these scores as the baseline scores. The DQA tool was designed to: (1) verify the quality of health facility data, (2) assess the system that produces that data, and (3) develop action plans to improve both [Cheburet et al., 2016]. We subsequently asked graduates to measure DQA scores from the same data source as the field project, except it should be 6 months beyond the baseline data. We then asked them to repeat the procedure at month 12 and finally at month 18 post-graduation. We used one-way ANOVA to determine if there were differences in the mean DQA scores over time.
DCA scores. The DCA is an end-to-end data integrity process. Because DCA focuses on the entire surveillance network, we did not ask the graduates to cross-check all data points in the system. We asked them to do a check of the indicators and counts used in their field projects. The first end is the generation of data at the health facility level. The middle is the county record, where the health facilities report their weekly and monthly tallies to the county health department (CHD) using MoH form 753. Then those data are entered into the district health information system (DHIS) by the county health records and information officer (HRIO). The DCA process is outlined in Fig. 2.
The goal is to detect inconsistencies as data travel through the surveillance system and identify root causes for these inconsistencies and to develop solutions, at the most granular level of the surveillance system -the health facility. The DCA score indicates the depth of the inconsistencies. A low score indicates high inconsistencies, whereas a high score indicates low inconsistency levels of the data between the 3 repositories of surveillance data. We used one-way ANOVA to determine if there were differences in the mean DCA scores over time. 7 Timeliness of reporting. Timeliness is a key performance measure of public health surveillance systems. Timeliness can vary by disease, intended use of the data, and public health system level.
The participants, as part of their field projects, had to evaluate the timeliness of reporting for the condition, disease, or health priority that was the focus of their field project. We used the results from the field project as baseline OTR measures. Then we followed up at 6 months to assess the proportion of reports submitted on time for the previous quarter. We repeated the procedure at 12 months postgraduation and then a final query at least 18 months post-graduation to examine the proportion of OTRs for the prior quarter. We used one-way ANOVA to determine if there are differences in the mean OTR scores over time.
Semi-quantitative measures. At the beginning of each training course, we asked participants to score their knowledge and skills in 8 key competencies on a Likert scale from 1 to 5, with 1 representing limited knowledge/skills and 5 representing expertise (Table 3). At the end of the 3-month training, we asked them to gauge their knowledge skills in each of those areas now that they have sat through 30 hours of didactic training, received hands-on coaching and mentoring from FELTP faculty, and completed a 5-week field project. We use the pre-post difference as our comparison point when we followed up after 18 months and asked them to rate their knowledge and skills now in terms of practical applications to their day-to-day work. We also asked their supervisors and colleagues to score the graduates' skills and knowledge and practical application in each of those competencies.
We used those scores to gauge the impact of FETP-F training on knowledge, skills, and change in work methods. We conducted a one-way ANOVA to determine if there was a difference in the scores between the 3 groups. Qualitative measures. The qualitative portion of the evaluation used grounded theory to determine the impact of the training, mentoring, and supervision on behavior, work method, and application of training to work duties [Reeves et al., 2015]. The grounded theory approach allowed us to develop our inquisitive instruments and then draw theory from them as we analyzed the interview transcripts, the interviewers' field notes from observations, and the previous responses to other evaluation tools as the graduates went through the 3-month training.
Semi-structured interviews were conducted with randomly selected graduates from groups 1-6.
Because we wanted to examine the impact of the training at least 1.5 years post-graduation, so that we could, at most, look at the first 6 groups to go through the FETP-F process. Those groups enrolled between July 2014 and July 2015. All interviews were recorded with the consent of the interviewees.
All interviewees had to provide written and verbal consent to the interview.
Participant observation. Participant observation allowed the field investigators to establish rapport with the person being interviewed so that the interviewee would provide more honest answers and opinions (vs answering what they think the interviewer wants to hear) [Laurier, 2016]. We used a checklist for field workers to note the presence of monitoring charts, active use of the DQA and DCA

Results
Demographics of survey respondents. For the quantitative analyses, 103 graduates were included in the analyses (Table 4). Most (55%) were male and 60% (n = 62) had < 10 years of public health work experience. Of the geographical regions, 36% were from central region (groups 2 and 5); 23% were from Nyanza region (group 4); 8% were from northern regions (group 3); 5% were from coast (group 6), and 5% were from the National Public Health Laboratory (group 1). The break down by cadre were 20% medical officers, 15% veterinary officers, 25% public health officers, 15% laboratory, 15% nursing, and 10% other. showed an increase in the mean DQA score from 75.64% at baseline to 84.53% at 18-months postgraduation (Fig. 3). This shows a 10.5% improvement in the mean DQA score for this sample of health facilities and programs. The subsequent ANOVA analyses on the 103 respondents showed that although the improvement was only 10.5%, that it still represented a significant improvement in DQA mean scores since baseline (Table 5). DCA scores. Descriptive analyses of DCA scores showed that there was an 11.4% improvement in DCA scores between baseline and 18 months post-graduation. However, upon further analyses using ANOVA, results showed that the increase was not significant (Table 6). Table 6 Comparison of pre-post score differences among the graduates, their supervisors, and their colleagues. Colleague  Statistics  1  2  2  Epidemiology  3  2  3  Surveillance  2  2  2  MS-Excel  1  3  2  Data analysis  2  1  2  Field investigations  2  2  2  Data audits  3  2  3  Communicating PH data 2  2  0 The scale is Likert, 1-5, with 1 = minimal knowledge/skill to 5 = exceptional knowledge/skill in the competency. Difference scores were calculated by subtracting the "pre" score from the "post" score. All results were positive. Between-groups: F = 0.76, f-crit = 2.90; p = 0.52.

Self Supervisor
OTR proportions. We examined the proportion of monthly reports submitted on time from health facilities to county health departments for the preceding quarter ( Table 7). The descriptive analyses show that there was a 60% increase in OTR between baseline and the 18-month assessment. The ANOVA showed this to be a significant development and improvement compared to baseline values ( Fig. 4). The ordinal scale ranged from 1 to 5 (1 = no knowledge, 2 = little knowledge, 3 = average, 4 = good, and 5 = mastery). SD = standard deviation; pre-assessment, mean scores before the training; post-assessment, means scores immediately after completing the 3-month training process; follow-up, mean scores at least 18 months post-graduation from FETP-F. Between-groups: F = 30.02; f-crit = 3.47; p < 0.0001. FETP-F = field epidemiology training program-frontline; SD = standard deviation; PH = public health Semi-quantitative self-assessment of learning scores (pre-post difference) compared to assessment by supervisors and colleagues showed significant increases (Table 8). Knowledge/skill levels within the 8 competencies were relatively low before the training. After training, we noted significant increases in the mean knowledge/skill scores in each of the 8 competencies. During the site visits, field workers also interviewed supervisors of the graduates and at least one colleague regarding any notable changes (positive or negative) after the graduate resumed his/her normal work duties. We used the same assessment scale as with the graduates. Results are outlined in Fig. 5. Table 8 Processes of analyses of qualitative data, n = 38 Process of open coding of the transcript to reduce the qualitative data to a more manageable focus. 2 We created categories from the level 1 open codes. This means that multiple level 1 codes were lumped together to create level 2 codes. 3 We then re-examined the level 2 codes to look for patterns, key words and "intent" of statements to generate themes and dimensions. This helped us tabulate the data and prepare it for subsequent analyses, such as inclusion in hierarchical regression models. 4 Reviewed the codes, themes, and dimensions to generate theory regarding the why and how (or why not) the training had an impact on its graduates and their affiliated organizations There was not much variation in the self-assessments of the graduates when compared to the assessments of competencies provided by their supervisors and colleagues. However, the supervisors and colleagues noted a marked increase in MS-Excel skills knowledge and expertise post-graduation.
For the larger group of graduates (n = 103), we examined via online survey the mean skills and knowledge changes (pre-post) in the key competencies before training (pre-assessment), immediately after the 3-month session ended (post-assessment) and 18 months after training (follow-up) (n = 103) [ Table 9]. Table 9 Results of analyses of the transcripts generated by interviews with the FETP-F graduates (n = 19), their supervisors (n = 12), and their colleagues (n = 7). • Then I also love the way we were put to task on how to present public health data • I have a lot of confidence; I like where people do presentations and I will really feel to be very keen because I can interrogate data also, I got some skills of just trying to see somebody's data and get a question out of one at least.
• I think more of it is coming to the projects that we had. The projects that we had -actually from what we learnt, it wasalmost after the basic training, there was an outbreak in Muranga for cholera. And because we had undergone that training, we were able to handle the case and we were able to isolate -to do the cultures and isolate and meet the identifications of the organism causing cholera. So, we were able to handle that outbreak within the county.
• Yea, because previously I would not know how to do the data analysis and also how to do thehow to write a paper, but now I have the confidence even to do a paper and also present it in the forums.
• Yes, like world bank we have been taking data from the…other facilities we have also been taking surveillance and given them a feedback and due to this I have also been going to other countries like Tanzania doing epidemiology and also collecting data and giving it to the world bank. Organizational • Data management improved • On time reporting rate improved • Improved response capacity to deal with repeated cholera outbreaks • Agency can make scientifically informed decisions about interventions • I feel she is prepared because ….as we have said, she has the knowledge now and let me site the same example, that of cholera. She was able to act very fast because the case was reported at around 4 pm and by five she was already here at the facility and she gathered everybody here and we were able to manage the same.
• She is better prepared, the reasons I had said is that what happens is that she is able to pick weekly data for -like we have diseases that we focus ion so much especially diarrhea infections and we have diarrhea much especially diarrhea infections and we have diarrhea prone areas. So, the facility catchment areas in those areas, she is able to tell us that this facility is now reporting abnormal numbers. Then she is also able to tell us-to sound a warning if those data are not real data. Because you know she is even the surveillance coordinator. graduates, n = 12 supervisors, and n = 7 colleagues). After transcription, we conducted 3 levels of analysis. The coding process was iterative and involved multiple stages that prepared and formatted the raw data so that they are available for evaluation. Each level of analysis is outlined in Table 10.
The results of 3 levels of analysis are outlined in Table 11. After conducting the level 1 analyses using key word searches and generation of word clouds, we had a list of 107 codes. During the 2nd level analyses, we reduced this to 37 codes that we later grouped into 25 themes. After the 3rd level review, we noted that the themes fell into 3 key dimensions. Graduates, their supervisors, and their colleagues' comments were associated with the "personal" (benefits to self), organizational (benefits to the agency or organization where the graduate worked or health partners in the graduate's community), and the FETP process itself (feelings and perspectives on the nominations/selection process, the execution of the course inclusive of its contents; and feedback on the quality of the 16 faculty and facilitators).

Discussion
This evaluation results provide support for the effectiveness of a localized field epidemiology and data management training process for improving skills and capacity of frontline health workers. During the interviews, most graduates, their supervisors, and colleagues reported that the course had helped them to make scientifically based decisions and improved their overall capacity to deal with a spectrum of public health challenges, from calculating thresholds to responding to cholera cases.
Additionally, they report that the course helped them to become better leaders by improving their communications skills, more evidenced-based decision making, and showing colleagues how to practically interact more critically with the data they generate at their agencies. . We also don't know the spectrum of participants' involvement in support networks, how the doctors' and nurses' strikes affected outcomes, the role of politics in who is nominated to participate in the training, local rates of job turnover, and how that the FETP-F does not award a diploma affects uptake among some younger public health workers.

Ethics approval and consent to participate
Informed consent was obtained from all FETP-F graduates who agreed to an evaluation visit. Personal identifiers were not included in the recorded data. Permission to conduct this evaluation was sought from and granted by the Ethical Review Board of the Ministry of Health (FAN: IREC 1795). This evaluation did not involve any animal subjects. The evaluation did not collect human subject data nor any human specimen samples. All subjects provided signed and oral consent for participation.

Consent for publication
Informed consent included consent to publish findings of this evaluation research. This research did not use any images, names, or other identifying information of any of those who consented for interview and participation in the evaluation. Therefore, a consent for publication was not needed from any of the research subjects.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.