Evaluating the effectiveness of in-service training program of Amhara Public Health Institute Dessie Branch, Northeast Ethiopia: a concurrent nested mixed quantitative/qualitative facility based cross sectional study

In-service training programs should be evaluated and modied regularly to provide quality health services. However, in Ethiopia, there is no published evidence about its effectiveness. Therefore, we evaluated the effectiveness of in-service training program of Amhara Public Health Institute Dessie Branch (APHI) using the Kirkpatrick model. In October 2019, a concurrent nested mixed quantitative/qualitative, facility-based cross-sectional study was conducted among 107 stakeholders from 22 randomly selected east Amhara government health facilities. The qualitative part also involved all this key stakeholder interviews. Data was collected using a semi-structured questionnaire through face to face interview. Epi data manager and Microsoft Excel 2016 software’s were used for data entry and analysis respectively. The major qualitative ndings were narrated and summarized based on thematic areas to supplement the quantitative ndings.


Introduction
In-service training is one of the commonly used strategies to improve health workers knowledge, skills and behaviors on new health approaches and technologies [1,2]. Its main objective is to capacitate health workers with the ability to deliver quality health services to all segments of the population [3]. To meet its objective, periodic evaluation of a training program could be conducted to identify major program strengths and weaknesses [4]. This helps training organizers to learn from their previous experiences; to know which stages of the training were successful and which not, or whether the approach to the training should be changed or not [5,6]. After all, the results of evaluation should be used to modify the contents and conducts of a training program [4].
In Ethiopian healthcare setting, laboratory professionals are responsible for performing routine as well as highly specialized tests to diagnose and/or aid in the treatment of diseases, preventing and solving problems with results, specimens, or instruments and communicating test results to the pathologist or clinicians [7]. Therefore, they need to be trained on specialized instrumentation and techniques to analyze patients' samples, such as blood, urine, stool, body uids, tissues and cells that might indicate cancer or other diseases [8]. Thus, the Amhara Public Health Institute Dessei Branch (APHI) in collaboration with partners is providing different capacity building trainings to laboratory and other health professionals in Amhara region.
However, training by itself cannot solve structural, organizational, or policy problems within an organization, although supportive supervision and the use of motivational strategies can help sustain performance improvement derived from training. The training evaluation also allows the identi cation of factors that restrain or contribute to a better performance of the trained health professionals, suiting the training actions to the organizational needs [9,10].
In Ethiopia, health care organizations including APHI spend substantial amounts of time, effort, and funding on quality of care improvement trainings, yet there is no published evidence about the effectiveness of in-service trainings. Therefore, evaluation of in-service training programs of Amhara Public Health Institute Dessei Branch is found to be important as it can provide information on the effectiveness of training services that would alert to plan for enhanced quality of trainings, give attention for better training programs. Also it highlights the major strengths and weakness in the training stages and the trainer institute.

Methods Of Evaluation Evaluation design and period
In October 2019, a concurrent nested mixed method, facility-based cross-sectional study was conducted to assess the effectiveness of laboratory in-service training program of APHI.
Study area and setting: Amhara Public Health Institute Dessie Branch (APHI) was established in 1982 as a regional laboratory research center and renovated as a branch public health institute in 2016. It is located in Dessie town, eastern Amhara, 401 km far from Addis Ababa (the capital city of Ethiopia) to the northeast direction and 490 km from Bahir-Dar (capital city of Amhara regional state). The institute has three main directorates; namely laboratory, research & technology transfer and public health emergency management [11].
The laboratory directorate provides technical support to health facilities found in the six zones, serves as in service training site for capacity building trainings, referral and specialized laboratory test site, perform public health lab for epidemic prone disease, external quality assurance activities, mentorship and accreditation. In these regard, the institute has the mandate to establish and maintain a high quality and sustainable laboratory system throughout the sub region. It also delivers quality and accessible laboratory services related to the occurrence, causes, prevention and diagnosis of major diseases of public health importance and to establish and support National Laboratory Quality Assurance programs and systems [12].
To realize it mission, the institute bene ts from the generous support of the Ethiopian and regional government and various national and international organizations. Some of the organizations that support the institute includes the World Health Organization through its Global Fund, Centre for Disease Control, Japan International Cooperation Agency, International Center for AIDS Care and Treatment Programs (I-CAP) and various non-governmental organizations through joint projects.

Sample size and sampling technique
Firstly, out of 30 external quality assurance (EQA) sites, 22 were randomly selected to represent the sub region. Then, ve laboratory professionals from each health facilities were randomly selected, making a nal sample size of 110 respondents. The qualitative part also involved all this key stakeholder interviews. All participants were laboratory professionals with a minimum of one-year work experience and who had ever attended on at least one laboratory in-service trainings in Amhara Public Health Institute Dessie Branch within the past two years.

Data collection
Data was collected using a semi-structured questionnaire through face to face interviews. The questionnaire had both open-ended and close-ended questions. Respondent's opinion, suggestions and comments were received through key stakeholder interviews. The questionnaire covered information on laboratory training pro les, training content and tool, training set up and training impact. Our respondents were laboratory personnel's from selected health centers and hospitals. The data collectors were the investigators themselves.

Data quality assurance
To minimize the subjectivity of responses from stakeholders, the investigators themselves participated as data collectors. During the session of each visit, we briefed the respondents about the purpose of the assessment, which was to evaluate the effectiveness of in-service training program of APHI; not merely the individual's performances. In addition, respondents were asked to feel open and comfortable to give their opinions, suggestions and comments that can help to improve in service training programs and future laboratory services.

Data analysis
All responses were graded as 'strongly agree, agree, not decided, disagree, strongly disagree and yes or no, except for the open-ended questions. Data completeness was checked on daily basis and the quantitative data entry was done through Epi data manager software and exported to Microsoft Excel worksheet 2016 for analysis. Data analysis was done at four levels, i.e. (i) Trainees reaction towards training, (ii) Trainers knowledge on the course & their communication skills, (iii) Learning of training content, (iii) Changes in job behavior after attending training (iv) Training set ups, bene ts and nancial issues. Descriptive statistics were computed to describe the data. Then the major qualitative ndings were narrated and summarized based on thematic areas to supplement the quantitative ndings.

Operational de nitions
In-service training Different, short term, practical training courses given by APHI for laboratory professionals working at health facilities to scaling up their knowledge and skills.

Training effectiveness
For each levels of evaluation, effectiveness is judged as not effective when values fall less than 50th quartile, partially effective between 50th and 75th quartiles and effective above 75th quartiles.

Results
Socio demographic characteristics of respondents.
A total of 107 laboratory professionals who were working in 22 selected health facilities (18 hospitals and 4 health centers) were interviewed which makes the response rate 97.3 percent. The proportion of male respondents was 60.7%. The age of the participants ranged from 20 to 49 years; 42% were between 25 and 29 years old. Majority of the respondents are diploma holders (53.3%) while nearly 44% are rst degree professionals. Regarding respondents work experience, 44% has one up to three years' experience whereas 34.6% of them had more than ve years work experience and the rest 21.5% are in between three and ve year experiences. Majority (84.1%) of the respondents are working in hospital and the rest 15.9% are in health centers (Table 1). 8.5%, and 6.5% of laboratory professionals were trained on dry blood spot/viral load sample collection and basic microbiology courses respectively.
Additionally, the qualitative ndings showed that training opportunities were almost equally spread across the health facilities and the number of trained professionals seems adequate. However, few of the respondents believed that there was trainee selection bias or selection criteria are not fair from facility to facility and also the calling modality was not appropriate, particularly at zonal health department level. proportion of respondents (80.4%) strongly agreed/agreed that the course provided opportunities to practice and would improve their job performance. Moreover, majorities of the respondents (86.9%) strongly agreed /agreed that the courses outcome was successfully achieved compared to meet their general expectations of the training and the courses gave trainee a clear understanding of the goals and objectives of the training before they started the training (96.3%).
In agreement with the quantitative nding, most of the respondents in the qualitative in-depth interview believed that the course structure, content and training tools of training were clear and understandable; courses providing opportunities to practice and the contents were in depth enough. Similarly, the availability of laboratory equipment's and tools has been appreciated, especially in the context of Strengthening Laboratory Management Toward Accreditation and Automation trainings. Nevertheless, some of the respondents raise "too short practical session and shortage of necessary machines and equipment's" as a problems regarding course structure, content and training tools. Trainer's knowledge and skills during training the course The proportions of respondents that strongly agreed/agreed on trainer's knowledge about the course content, their communication and delivery skills were nearly similar. Hence, 91% of respondents strongly agree or agree that trainer's be able to deliver training course's properly and majority of respondents (88.8%) strongly agree or agree that trainer's communication skills was able to provide all the information needed for the course. About 86% of respondents strongly agrees or agrees regarding trainer's knowledge on the course. Yet, 13.7% of respondents strongly disagrees or disagrees with that of trainer's ability to create participatory and interactive training environment; though 12% of the respondents are not able to decide on this issue.
Findings from the qualitative part also revealed that majority of respondents acknowledged the trainer's knowledge and skills as a potential to con dent the trainers. Others also reported more positive notions of knowledge, skills and their willingness to train based on their praising. In contrary to these, as stated by few of the respondents, some of the problems related to knowledge and skills of trainers includes; undermining of trainee, a few trainers are aggressive, sometimes trainers are none expertise or nonlaboratory professionals, few trainer's presentation style is lecture so better to be entertaining style, trainers did not took training of trainers.  At this level of evaluation, 93.5% of trainees mentioned as if they are applying the knowledge and skills they gained during the training in their workplace and 69.2% of them tried to qualify other laboratory professionals in their working areas, 68.2% of them are using the training materials properly. However, 73.8% of the respondents have replied that there is no registration system that indicates trained staffs in the lab unit of their facility and 55.1% of them also indicated that they didn't assign trained persons to appropriate positions. Training set ups and environment More than half of the respondents (51, 4%), strongly disagreed/disagreed with the training set up (hall, toilet, café) suitability for the training. In the meantime, 46.7% respondents strongly disagreed/disagreed with accommodation perdeim and or refreshments of the training program. As well, 28.9% of the respondents mentioned as if the calling modality for the training was inappropriate. Still, 69.2% of the respondents strongly agreed or agreed on time and season of the training.
The nding from the qualitative part also nd out that many of the respondents had a complain concerning training set ups, bene ts and nancing processes. Thus many of them have been ll discomfort with training hall, toilet, café, nancing process, insu ciency of perdiem, and not happy with tea and snacks. As well, some respondents have been disgusted by extended waiting times, guilty words and insults while processing their accommodation perdiem. Overall, the authors have tried to summarize the effectiveness status of each levels of training evaluations. As a result, participant's reaction and their learning events were rated as effective; while post training behavioral change and the training set ups were partially effective. In general, the average cumulative scores of each level of evaluations fall in the fourth quartile (76.2%). Thus, according to our operational de nition, this gure shows as the in-service training program of APHI was more or less effective (Table 8).

Discussion
This study has tried to assess the effectiveness of Amhara Public Health Institute training programs using the Kirkpatrick models of training evaluation. Thus, in the rst level of evaluation, our participants were motivated and reacted positively on the course structure, content and training tools, which has also documented in a study from Egypt [13]. Our nding at this level might indicate excellences in training module preparation, training need assessment and trainee recruitment.
Regarding trainer's knowledge and skills on the course content, most participants point out that trainer's knowledge about the course content, their good communication and delivery skills were adequate to provide all the information needed for the course. This indicates the effectiveness of "training of trainers" program that quali ed trainers with the technical requirements of a good trainer. Besides, allocating adequate number of trainers with su cient training time, will allow to entertain different ideas in a classroom which provides more discussion opportunities to trainees to acquire all the knowledge and skill competencies of the training. Conversely, the allocation of non-competent trainers in the present study could be attributed with trainer's knowledge gap and/or lack of expertise or intentionally assigned by training organizers; just for the seek of individual bene ts or due to lack of experienced trainers. These comments may indicate a weakness in providing training to address actual training gaps.
In the second level of our evaluation, majority of our participants really stated an improvement in their knowledge and skills after training. In agreement to this, a study done in Sweden and Sri Lankan also pointed out the effectiveness of training in improving trainee's knowledge and skills [10,14]. Moreover, another systematic review emphasized the importance of training to health professionals in order to enrich their pre-existing knowledge and skills [15].
Moreover, our evaluation at the behavioral level asserted that the knowledge and skills they gained during the training has positively impacted their work performance in majority of the cases. This result is also evidenced in a recent systematic review on essential newborn care training that suggests improvements in delivery room newborn care practices as compared with usual care [16]. In this phase, our nding implies the success of APHI training programs to achieve its goal deeming that the success at behavioral level will leads to result.
Furthermore, nearly half of the trainees declared that the training environments were not suitable and appropriate for learning. Thus many of them have been ll discomfort with training halls, toilets, café, the nancing process and insu ciency of perdiem, and not happy with tea and snack refreshments. The unfavorability her is partially explained by the fact that the training institute is currently working in a temporary and non-standardized building blocks with con ned and limited classrooms. Unless otherwise these kinds of problems are identi ed and resolved early, they will tend to grow and amplify themselves to the extent of causing a signi cant negative impact on the organizational achievement [4].
A limitation of the present study includes the lack of an observational part of the trainees practice and performance in their working health facility that usually affects quality of tests in the laboratory. On the other hand, this study did not address the result level of Kirkpatrick model that investigates the total training costs against it bene ts. It shows the need to study further at a deeper level to get a realistic picture of APHI training program effectiveness.

Conclusion And Recommendations
Generally speaking, the laboratory in-service training services in Amhara Public Health Institute were marginally effective with its own limitations. To meet its overall organizational objective, the effectiveness of training programs could be regularly monitored and evaluated based on a standard set of training performance indicators. Furthermore, training organizers should utilize lessons learned /feedbacks in this study to create a conducive and appropriate environment for learning which improves the quality of training service.
Abbreviations APHI: Amhara Public Health Institute Dessei Branch

Declarations
Ethical approval and consent to participate Ethical approval was obtained from Institutional Review Committee of Amhara Public Health Institute. Formal letter was given to each health facilities to obtain permission and cooperation. After brie ng the purpose of the study oral consent was obtained from each participant.

Consent for publication
Not applicable Availability of data and materials All relevant data are within the manuscript and its Supporting Information les.

Competing interests
We declare no competing interests.

Funding
Funding was not received from any organization, but APHI has covered data enumeration cost.
Authors' contributions SL designed the study, participated in and monitored the data collection process, analysed the data, and wrote the rst draft of the manuscript. TA participated in the design and analysis of the study. He also involved in rst draft writing, reviewing and modifying the drafts of the manuscript. The rest author's had equal contribution in the design and data collection stages of the study. All the authors have read and approved the nal version of the manuscript.