Long-term outcomes of a program to upgrade the nursing faculty in Cambodia: a qualitative study

Background Well trained nurses are expected to inuence their workplaces after completing their training courses. In Cambodia, a high-skill training program has been given for nurses to make them more qualied since 2011. After completing the program, we conducted an immediate post-training study in 2014. In this long-term, post-training study, we evaluated the program’s effect on nursing faculty and clinical preceptors and identied factors inuencing the long-term outcomes. Methods This long-term, post-training study took the form of embedded, single-case design. It incorporated focus group discussions (FGDs) for 26 trainees, key-informant interviews for 11 institutional managers, and teaching document reviews. Then, we conducted a thematic analysis using Kirkpatrick’s four levels of training evaluation. We evaluated the latter two elements as long-term outcomes, behavior (changes in trainees’ practice and applying learning to practice), and results (the trainees’ behavior effect on trainees’ organization or department). After that, we assessed factors inuencing the long-term outcomes using Bronfenbrenner’s ecological systems theory. Results We identied ve themes at the behavior level according to program type. They were 1) improved teaching and learning activities, 2) built capacity of nursing faculty and clinical nurses, 3) improved clinical practicum, 4) strengthened clinical nursing management, and 5) involved in professional development activities. At the results level, we identied three themes as long-term outcomes from an institutional development aspect. They were 1) enhanced institutional capacity, 2) improved students’ learning and behavior, and 3) strengthened clinical performance. The major inuencing factors for the above eight themes were 1) institutional managers’ support, 2) continuous networking among trainees, and 3) national policy. The upgrading program had a favorable effect on nursing faculty and clinical preceptors at the Kirkpatrick model’s behavior and results levels. Compared with the immediate post-training study ndings, the perceived long-term outcomes became more strategically focused and demonstrated the training impact on a considerably broader scale within the workplace. These ndings have implications for the and of nursing faculty for its

the Kirkpatrick model's behavior and results levels. Compared with the immediate post-training study ndings, the perceived long-term outcomes became more strategically focused and demonstrated the training impact on a considerably broader scale within the workplace. These ndings have implications for the monitoring and evaluation of nursing faculty development programs for its sustainable outcomes Background Building a competent health workforce is critical to accelerating the achievement of Universal Health Coverage [1]. As upgrading nurses' academic backgrounds are vital to improving clinical outcomes [2], nursing education has put more emphasis on a higher degree-level program [3].
To achieve the optimum outcomes of higher education programs, institutional development of education, and clinical facilities is also crucial. It can provide an environment in which more educated nurses can work better. However, it is challenging to create a mechanism to ensure how individual learning and training outcomes can be well applied in education and clinical workplaces [4]. In Asia, due to many contextual factors, trained nurses cannot easily use their knowledge and skills after coming back to their workplaces in education and clinical facilities. These include obedience to hierarchy, respect for seniority, lack of consensus among faculty members in the workplace, and dependency on personal commitment [5].
To evaluate nurses' training programs, Kirkpatrick's model is well known, which describes four levels of outcomes; 1) reaction and satisfaction, 2) learning, 3) behavior, and 4) results. The rst level evaluates trainees' immediate reaction to evaluate the degree to which participants nd the training favorable, engaging, and relevant to their jobs. The second level evaluates the degree to which participants acquire relevant knowledge, skills, and attitudes. The third level evaluates the degree to which participants apply what they learned during training when they are back on their job. For the last level, it evaluates the degree to which targeted outcomes occur in the broader area of work as a result of the training [6,7].
According to a systematic review of faculty development, more research has been done about reaction and learning than behavior and results. In particular, the evaluation of results remains relatively unexplored, such as changes in organizational systems and changes in students [8].
Cambodia's faculty development program Agency (JICA). Thirty-two nursing faculty and clinical instructors were trained in four cohorts [9] between 2011 and 2015.
In our immediate post-training study performed in 2014, we interviewed 34 nurses (55.7 % of 61 trainees) and 10 managers from 8 participating agencies (57.1% of 14 participating agencies) sometime after 6 months to two years of "upgrading" through one of two types of programs. According to Kirkpatrick's model, we assessed their 2) learning and 3) behavior and found the bene ts with both professional nursing practice and teaching after completing the upgrading courses. However, the immediate post-training study focused only on individual behavior outcomes, and we found these trainees were not well accepted by their nursing colleagues and other health professionals at their workplaces [10]. In this longterm, post-training study, we aimed to evaluate the 3) behavior and 4) results as long-term outcomes of these two training programs and identify in uencing factors by emphasizing institutional development of education and clinical facilities.

Methods
We conducted an embedded, single-case, qualitative study. First, we conducted key-informant interviews and focus group discussions. Second, we reviewed teaching documents to triangulate the ndings.

First step
We collected data from two different groups (Table1). The rst group consisted of 26 trainees who are nursing faculty members and clinical preceptors from four schools and ve hospitals. They completed the upgrading program from 2012 to 2014; hence, data were collected some 4 to 6 years since they had completed their training program. All trainees who participated in the immediate post-training study in 2014 were included in this long-term, post-training study. This group comprised (1) eight nursing faculty members who participated in an in-country program, (2) eight nursing faculty members who attended a foreign-country program, and (3) ten clinical preceptors who attended the foreign-country program. The second group comprised 11 managers from nine participating agencies. They held positions such as school directors, technical bureau chiefs, hospital directors, and nursing directors. They supervised the faculty members and clinical preceptors described as the rst group. To identify the long-term outcomes from an institutional development aspect, we conducted key-informant interviews or focus group discussions by the institution.
We developed a topic guide and conducted semi-structured interviews at the participants' workplace. In this long-term, post-training study, we assessed Kirkpatrick's model's two elements, 3) behavior and 4) results. Table 1 lists the areas of inquiry for the topic guides as follows: postgraduation activities based on acquired skills from the upgrading program (behavior evaluation); the program's outcomes at institutions (results evaluation); in uencing factors (e.g., opportunities and challenges within and outside of the workplaces); and the request to further improve the nursing professionals in the topic guide.

The interviewers (KKS and MS) conducted interviews with a native bilingual interpreter uent in Khmer
(the Cambodian language) and English from April to May 2018. We pilot-tested the topic guide to con rm its face validity and accuracy of interpretation. We audio-recorded all interviews. An author and research assistants took eld notes throughout the discussion with the participants' permission.
The research assistants who joined in the interview transcribed the audio recording in English. We went through the transcripts to detect any discrepancies from our eld notes. Two of the authors (KKS and NF) read the transcripts thoroughly and interpreted the data. We conducted a thematic analysis to generate coding from phrases that corresponded to program trainees' behavior, results, in uencing factors, and further development. We then reviewed them repeatedly. We developed categories and subcategories until none others were apparent. After producing the themes, we compared the themes between nursing faculty, clinical preceptors, and institutional managers. To obtain trustworthiness, we followed the steps outlined by Nowell [11]. We conducted qualitative data analysis by manual-coding using Microsoft Excel 16.3 (Redmond, Washington, USA).
Furthermore, we categorized the in uencing factors according to the ecological systems theory. The Bronfenbrenner's ecological system theory explains how human development is in uenced by different environmental systems [12]. It consists of the following four levels: (1) microsystem: individual factors, (2) mesosystem: organizational or institutional factors, (3) exosystem: local context or community, and (4) macrosystem: national-level factors [13]. The rst author (KKS) presented the summary of ndings to the participants' representatives and received valuable comments to re ne the ndings further.

Second step
The rst author and research assistants visited three public educational institutions. The interviewed nursing faculty worked and collected teaching and learning documents from them (June to August 2019).
The research assistants and the rst author classi ed the translated version according to the teaching documents. They then compared them with the themes from interviews for data triangulation (Additional le 1) [14].
Before data collection, we obtained verbal consent from each participant and assured that all personal information would remain con dential.

Results
General characteristics of participants The in-country program's nursing faculty had an average of 5.5 years more teaching experience than those who had attended the foreign-country program. There were more female participants among the clinical preceptors. Table 2 shows participants' demographic information.

Evaluation of long-term outcomes
Thematic analysis identi ed ve main themes in the behavior level and three themes in the results level.

Evaluation of behavior
Improved teaching and learning activities The nursing faculty worked in advancing the following teaching and learning activities: developing course syllabi based on students' evaluation criteria, preparing case scenarios to teach the nursing process, applying student-centered approaches to classroom teaching, and preparing lesson plans among the faculty's nursing unit.
"Before, we used only to give a lecture. Now, we offer more time for students to exercise their activities, research, and work on their assignments." (Nursing faculty from the in-country program 2) Built capacity of nursing faculty and clinical nurses Both the nursing faculty and clinical preceptors contributed to building their fellow faculty and clinical nurses through educational activities such as training, conferences, and workshops.
"The trainee conducted training and education to their colleague clinical nurses to the points when he observed during supervision at the hospital, such as hygiene and infection control." (Hospital manager 3)

Improved clinical practicum
The nursing faculty and clinical preceptors who learned in foreign-country programs worked on advancing the clinical practicum. They developed a case assignment form for students and provided training to clinical preceptors regarding the nursing process.
"CMoH provides a workbook that concentrates only on the clinical nursing techniques. We thought this was not enough; students needed to put the theory into practice. Therefore, we provided case assignments for students." (Nursing faculty from foreign-country program 5)

Strengthened clinical nursing management
Clinical preceptors contributed to improving nursing management, reinforcing the nursing department's function, improving shift handover report, and modifying the ward's working shift. They also developed the nursing policy and nursing protocol of the hospital.
"I was happy to work on a team of nursing leaders. The hospital allowed us to establish the nursing department for nursing activities. When we did not have the nursing department, nurses didn't know who was supervising us" (Clinical preceptor from foreign-country program 2).

Involved in professional development activities such as council and association
Many nursing faculty members and clinical preceptors were involved in professional improvement activities of the council and association. Those from the foreign-country program initiated an alumni association's establishment, which culminated in the Cambodian Association of Nurses in 2019.
"We now have a strong group of nurses and midwives who have always been together, and we discussed things very openly. Sometimes, in Cambodia, it was hard to talk about things openly. However, unlike in this group, we spoke the same language and understood the same things. We faced many challenges, but we were still together." (Clinical preceptor from foreign-country program 10).

Enhanced institutional capacity
Through the nursing faculty and clinical preceptors' capacity-building, we identi ed the following outcomes: improving collaboration among nursing faculty, and enhancing cooperation between an educational institution and hospital on a clinical practicum.
"Clinical preceptors were also interested in case assignments for students during clinical practicum. We explained it to them through the technical meeting." (Nursing faculty from foreign-country program 5)

Improved students' learning and behavior
The nursing faculty and their managers reported that the students' performance and behavior have improved. The in uencing factors were arranging the teaching and learning content and enhancing the teaching and assessment methods.
"The students were afraid to ask questions or answer the faculty's questions. Through encouraging students, they become more active and understand more." (Nursing faculty from in-country program 4) Strengthened clinical performance at the hospital Clinical preceptors and their managers reported improvements in clinical performance by reducing the workload, providing a better quality of care, enhancing communication among nurses, and promoting collaboration among health professionals. Consequently, patient satisfaction was achieved.
"Developing a nursing protocol was also bene cial to other departments. For example, a surgical department developed a nursing care protocol on wound care shared with the Intensive Care Unit. Therefore, both departments follow the same protocol. The patient did not have to be transferred to another department to receive wound care." (Hospital manager 3)

Factors in uencing the long-term outcomes
Factors in uencing the long-term outcomes were identi ed and categorized into micro-, meso-, exo-, and macrosystems according to the ecological systems theory. Figure 2 depicts such factors according to each system. At a mesosystem level, the managers provided the following supports: promoting a managerial position, supporting an activity nancially, and providing encouragement. The nursing faculty and clinical preceptors were given more responsibilities to improve teaching and learning activities. They worked as a team.
"I nominated trainees for promotion. They can contribute to the working group for teaching and learning activities, and their salary also increases by promotion." (School manager 3) At a mesosystem level, recognition from other health professionals (particularly medical doctors) was a promoting factor for clinical preceptors (printed in blue in Figure 2). The inhibiting factors were teaching and learning environments with defective materials and references (printed in green in Figure 2) for the nursing faculty. For both the nursing faculty and clinical preceptors, the inhibiting factors were the lack of support from a nurse colleague and their limited capacity.
"We had a faculty who didn't understand the nursing process well. In the unit, we discussed what nursing diagnosis means and how we could diagnose; we shared such information among faculty." (Nursing faculty member from foreign-country program 3).
At an exo-system level, we identi ed external supports such as the technical support from development partners and their faculties of upgrading programs. The nursing faculty and clinical preceptors who learned in a foreign-country program had more opportunities to go abroad for additional training or conferences. The collaborative networking among trainees was also identi ed.
"I consulted with the faculty in Thailand about reforming the nursing shift from 24 hours to 12 hours. It was a sound system, and it continues until now." (Clinical preceptor from foreign-country program 2). Finally, at a macrosystem level, strengthening the regulatory framework was a promoting factor. Some managers reported that CMoH's commencing the National Exit Exam in 2013 in uenced the school to begin to pay more attention to the quality of education.
"Because we focused on the quality of education, all of my students need to pass the National Exit Exam." (School manager 2) Meanwhile, the inhibiting factors were identi ed as follows: inconsistencies in teaching content between educational institutions and hospitals, outdated curriculum, and overburdened workload due to the task concentration. Trainees also frequently mentioned the failure to recognize a BSN in the payroll of the public servant system.
"Although I obtained a BSN, I still receive an ADN salary, which affects my motivation. The government did not give us the value of our academic background." (Clinical preceptor from foreign-country program 5)

Discussion
This long-term, post-training study identi ed ve main themes in the behavior level and three themes in the results level as long-term outcomes. Furthermore, the above eight themes' major in uencing factors were 1) institutional managers' support, 2) continuous networking among trainees, and 3) national policy.

Long-term outcomes at behavior level
We identi ed ve outcomes at behavior level: 1) Improved teaching and learning activities, 2) built the capacity of nursing faculty and clinical nurses, 3) improved clinical practicum, 4) strengthened clinical nursing management, and 5) involved in professional development activities. In the immediate posttraining study, the observable outcomes were directly related to the relevant professional's daily work, such as applying new knowledge to one's work in teaching and clinical practice and disseminating new knowledge to the colleagues [10]. Four to six years after completing the program, the behavior outcomes became more strategically focused. Knowledge and skills were being utilized considerably broader and more sustainable scale within the workplace. This nding supports that the training participants shared their knowledge with others and aimed to start activities that would build capacity in their home institutions [15]. Moreover, exploring nurse education and practice in a foreign setting was valuable in considering new ways of working, particularly identifying collaborative working relationships between education and clinical facilities [16].

Long-term outcomes at results level
We identi ed three outcomes for at results level: 1) improved students' learning and behavior; 2) enhanced institutional capacity, and 3) strengthened clinical performance at the hospital. In the immediate post-training study, the managers had higher expectations toward the upgrading program outcomes, which was the undergraduate level, such as preparing course syllabi. However, the results outcomes that trainees' behavior affects their institution involves a learning process. Thus, the outcomes may not immediately meet expectations from their managers [10]. This nding supports ensuring monitoring and evaluation in the faculty development program's design to enhance accountability for subsequent implementation [4] to maximize the sustainable impact of health professional training from an institutional development aspect [17]. The bene ts in such cases would accrue not only to those who participated directly in the program. Nevertheless, by others within their spheres of in uence -the educational and clinical care programs they contribute to, the colleagues and junior professionals with whom they have contact, and the institutions within which they operate.
Major in uencing factors for the eight outcomes The rst "stimulating" factor contributing to positive institutional bene t, was the type of managerial support operating at the mesosystem level. Strategic managers promoted trainees to responsible positions. In the immediate post-training study, the program trainees faced challenges in understanding the new concepts and methods, among the nurse colleagues and other health professions who had not been exposed to the training [10]. The support from managers is highly in uential in a society such as that in Cambodia, where obedience to the hierarchy and respect for seniority is essential [5]. This nding supports that the manager's strategy was hugely in uential in determining how best to utilize the institution's upgraded personnel [18].
The second factor was continuous networking among trainees. It provides them a sense of connection with people who share the same passion and values. This nding supports the value of longitudinal programs extending beyond teaching effectiveness, such as facilitating networks and collaborative meetings over time [8]. In the immediate post-training study, the trainees formed groups to lead strategic improvements in nursing education [10]. Notably, the group culminated in building a community of practice [19], that transformed into, the Cambodian Association of Nurses. The members who continue studying up to master and Ph. D. levels play a crucial role in their continuous professional development as nurse leaders' cohorts and inspire the younger generation of nurses. The ndings also demonstrate that the trainees are potential leaders contributing to the advocacy activities, bringing nurses' involvement in health policy formulation [20,21].
The third crucial institution-supporting dimension was the policy direction at the macrosystem level, which was in uenced strongly by the Association of Southeast Asian Nations (ASEAN) Mutual Recognition Arrangements (MRA). MRA stimulates the development of relevant laws and regulations towards establishing a mutually acceptable professional regulatory framework among member states. In Cambodia, some policies were developed as a consequence of signing the MRA, commencing the National Exit Examination, and increasing the number of BSN faculty [22]. The number of faculty members who hold BSN academic backgrounds increased from 10 in 2010 [23] to 68 in 2019 in public educational institutions (personal communication with educational institutions, 2019). It enabled the nursing faculty members to work as a team to improve a range of teaching and learning activities. This long-term, post-training study reinforces the value of developing a regulatory framework that in uences both the quality of care and pre-service education [24].
There is a possibility that a degree of selection bias may have occurred in this long-term, post-training study. The nursing faculty and clinical preceptors who were active in their workplaces may have been more willing to participate in the interview. The main interviewer was involved in the upgrading program supported by JICA; therefore, participants may also have sought to respond positively to a donor. Among the features that would have promoted effective engagement with the study was the long-term collaboration between the primary interviewer and the trainees, the study's construct validity, and the data triangulation from several sources.
For future research, the extent of the upgrading program's contribution to achieving the ultimate goals could be quantitatively evaluated. The ultimate goal of faculty development is to improve students' performance and learning and enhance the clinical outcomes of patients [25]. Also, identifying the most in uential factors that enhanced the program's success suggests the need for ongoing support after completing the upgrading program if the bene ts are to be maximized within an institutional context [4].

Conclusion
Nurse faculty development through an upgrading program for Cambodian nurse professionals yielded favorable outcomes when examined in relation to the long-term outcomes with respect to two elements of Kirkpatrick's model. The long-term outcomes of the trainees' activities became more strategically focused and demonstrated the knowledge and skills on a considerably used wider scale within the workplace. Institutional manager's support, continuous networking among trainees, and national policy, served as major driving factors to improve their institutions' teaching and learning activities. 2) was also attached.

Consent for publication
Not applicable

Availability of data and materials
The data that support the ndings of this study are available on request from the corresponding author KKS. The data are not publicly available due to them containing information that could compromise research participant privacy/consent. Evaluation for each level according to the type of participants.

Figure 2
In uencing factors of the long-term outcomes.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download.