Program Evaluation of a Nationwide Continuing Professional Development (CPD) Program in Korean Public Health and Medical Institutions

Background: Education & Training Centre for Public Healthcare of the National Medical Centre plays a key role in providing continuing professional development (CPD) to 221 public health and medical institutions in South Korea. To assess the realisation of the Center’s core value and of the intended chages, program evaluations is needed. The context, input, process, and product (CIPP) model is particularly suitable for evaluating CPD in public sector as it allows to recognise the dynamic nature of the programme environment. Methods: This research applied the CIPP model to the evaluation of CPD programmes, particulary abdominal and thracic ultrasond programme implemented in 2017 and 2018. Data were collected from 2017 to 2019. The programme and its feedback were reviewed in context evaluation. Based on this, the subsequent programme strategy was established in input evaulation. Observing the programme in real time and recording progress was followed in process evaluation. Finally, the outcome and impact of the programme were reviewed and compared with baseline data in product evaluation. Results: In context evaluation, the educational needs of the recipients of the Centre’s CPD programme, impediments that inhibit participation in education, and resources that the Centre can utilise were identied through online survey, focus group interviews, expert consultation. Through input evaluation, we identied the best alternative that satised all pre-selected criteria, which were responsiveness to priority system needs, potential effectiveness, t with existing services, affordability, and administrative feasibility. Observing the programme in real time and recording progress was conducted in process evaluation and shows that the improved programme went as planned, and even had to be expanded due to increased demand. The impact of the programme was measured, interpreted, and assessed in the product evaluation. As the review committee decided that the intended change has occurred, the Centre decided to maintain the program. Conclusions: Thorough evaluations are necessary to unlock the potential benets of CPD. The CIPP methodology is valuable to execute formative as well as summative evaluations. The CIPP model is especially useful for securing accountability data for large-scale nation-wide educational programmes supplied by public funds.


Background
Continuing professional development (CPD) for physicians is important to achieve safe, effective, and improved clinical care [1]. However, the potential of CPD programs is not fully realised as they tend to deliver arbitrary content rather than to meet practical educational needs, selecting teaching methods based on educator convenience such as lectures or observation. Much needed emphasis is now being placed on gap analysis [2], targeting practice change [3,4,5], and adult education practices [6]. Thorough evaluations of existing programmes are the rst step for applying innovative CPD approaches in educational practice.
Education & Training Centre for Public Healthcare at the National Medical Centre (NMC), hereafter referred to as the Centre, plays a key role in providing CPD to 221 public health and medical institutions (PHMIs) in South Korea. PHMIs are established and operated by the state, local government, or public institutions [7]. The Centre is responsible for providing CPD to 57 PHMIs that fall under the supervision of Ministry of Health and Welfare, comprising over a quarter of total PHMIs in the country and 2,121 physicians as of December 2018, making up 17.1% of the total number of physicians in Korean public healthcare hospitals. Due to its sheer reach, the Centre is one of the most in uential institutions in the public healthcare sector.
The Centre's core values are enhancing publicness in healthcare and improving the quality of medical services [8]. Enhancing publicness in healthcare means strengthening the role of public health and medical institutions to ful l unmet needs in the healthcare system, addressing areas where the free market typically fails (i.e., infectious disease outbreaks, natural disasters, or emergencies).
Programme evaluations should assess how well the Centre's core values are realised and whether the intended changes have occurred. Educational programmes are fundamentally about change, and therefore programme evaluations should assess whether desired changes occurred as the result of educational initiatives [9]. The CPD setting calls for speci c criteria that the programme must meet. For example, since CPD is funded by public money, accountability is particularly important along with potential effectiveness and administrative feasibility. In addition, it is important to assess whether the needs and objectives of the programme are clear, as well as whether the programme is tailored to meet these needs and objectives. The Centre's programmes are required to prepare physicians for public health roles regardless of their prior residency training or specialty, to be capable of treating the majority of patient groups including chronic disease patients, and to be prepared to meet public healthcare demand in times of disaster.
Among diverse methodologies [9,10,11], the CIPP (Context, Input, Process, and Product) model has the most suitable characteristics for the evaluation of CPD in the public healthcare sector, which is affected by external and internal factors as well as factor interactions. Situations such as the outbreak of new infectious disease or natural disasters are what physicians in the public healthcare sector need to be trained for.
They should also be able to respond to the increased need for treatment of chronic disease patients and issues like an aging population.
Diverse participants and stakeholders, governmental and related ministries that operate programmes, programme policies, and the call from civil society also affect educational programmes. These factors in uence one another. For these reasons, public health CPD should be treated as a nonlinear, complex, dynamic system [12,13].
Informed by complexity theory [14,15], the CIPP model recognises the importance of clinical and educational context and accommodates numerous uncertainties within the educational programme [9]. The CIPP model allowed us to recognise "the dynamic, septic condition of the real world" [16 p. 351], and at the same time meticulously take into account the programme environment and the multiple inputs to the programme, as is essential in the evaluation of public health CPD. Additionally, the CIPP model can provide both formative and summative evaluations [16,17], and therefore the model's strength is in monitoring implementation and measuring improvements during and after implementation, usually on a yearly basis. Moreover, as CPD programmes are funded by taxpayers, they must be held accountable. CIPP is particularly useful in this regard as it translates all data involving strategy planning, provision and monitoring of education as well as objective assessment into accountability data. Therefore, the CIPP model is suitable in the evaluation of CPD by the Centre.

Methods
We applied the CIPP model to evaluate CPD programmes implemented by the Centre in 2017 and 2018. The Centre offered 17 programmes in 2017 and 10 programmes in 2018. Among these, we evaluated the abdominal and thoracic ultrasound programmes because major changes were made to these programmes in terms of strategy, educational goals, content, and methodology in 2018, following a needs assessment in 2017, which showed the strongest demand for change among other programmes. Educational evaluation is useful for examining the effectiveness of a programme after changes are made and determining whether to continue the changes.
To answer evaluation questions in each phase of the CIPP model, data were collected from May 2017 to January 2019 (Table 1) from learners, instructors, operators, stakeholders, and external experts. We evaluated abdominal and thoracic ultrasound courses delivered to physicians at PHMIs working under the Ministry of Health and Welfare. This study was approved by the Institutional Review Board of National Medical Centre, Seoul, Korea. -Periodic exchange of information with project leaders and staff to monitor and provide feedback on the process and record the actual process Product What positive outcomes of the programme can be identi ed?
What negative outcomes of the programme can be identi ed?
Were the intended outcomes of the programme realised?
Were there unintended outcomes, either positive or negative?
What are the short-term implications of programme outcomes?
What are the longer-term implications of programme outcomes?
How effective was the program?
How sustainable are the intended and positive programme outcomes? We used the CIPP methodology as an ongoing evaluation tool. The ultrasound programme and its feedback were reviewed in 2017 through context evaluation. Based on this, the 2018 programme strategy was established through input evaluation. Planning and implementation of the programme were assessed through process evaluation. Finally, the outcomes and impact of the 2018 programme were reviewed and compared through product evaluation with baseline data from the context evaluation. Both formative and summative evaluations were executed using the CIPP model. The formative evaluation was executed by two of the authors, JL and HM, who participated as operators in continuous monitoring.
The summative evaluation was executed by all three authors, including CJK, who participated in record analysis as an external expert after the completion of the programme in both 2017 and 2018.
Educational needs, impediments, opportunities, assets, and resources such as expertise and services should be included in context evaluation.
To assess the educational needs of potential participants in CPD programmes, we conducted an online survey targeting 1,010 physicians at 39 PHMIs of whom 204 of replied, yielding 178 valid responses. We conducted focus group interviews (FGIs) from December 5, 2017 to March 8, 2018 to achieve in-depth interpretations of the survey results. In total, 106 valid responses were received from 111 participants.
FGIs targeting learners followed a democratic view of educational needs assessment approaches [18]. This approach is advantageous because it involves many stakeholders in goal-setting, allowing them to determine the relative importance of potential needs. However, as this approach may uncover participants' preferences rather than their actual needs, we did not solely rely on learner FGIs. The same research questions were posed to an expert advisory group so that possible shortcomings associated with FGI responses could be complemented by informed analytic views [18]. In addition, document review, demographic data analyses of learners, and 2017 programme record analyses were included in the contextual evaluation.
In input evaluation, the most cost-effective and feasible strategy was selected to best satisfy the needs identi ed through context evaluation.
Our online survey and FGI-based needs assessment indicated that the ultrasound programme was most in need of improvement. Therefore, we performed an input evaluation of the ultrasound programme. A literature review, expert consultation, and stakeholder workshop were used to identify strategy selection criteria [17]. Upon the completion of the 2017 programme, three external experts reviewed records, assessed outcomes and impacts, and made suggestions for improvement. Twenty-eight directors of public hospitals also participated in the stakeholder workshop and developed strategy selection criteria for the 2018 programme.
Process evaluation was conducted by providing an ongoing assessment of plan implementation, followed by documentation of the process [17]. Acting as process evaluators, two of the authors compared the initial plan for the programme and actual practice, identi ed the reasons for differences, and monitored feedback for each participant. Real-time monitoring and documentation followed.
Additionally, an open-ended survey was administered to learners, instructors, and operators after the completion of each programme module.
The goal of product evaluation is to measure, interpret, and assess the outcomes and impacts of the programme [17]. Upon the completion of each education module, participant satisfaction and reports of project effects on outcomes and impacts were surveyed. Academic achievement was assessed at the end of the module and compared with pretest results. An online survey assessing how well the learning was applied at work sites was conducted 3-6 months after programme completion. As a stakeholder, an education and training review committee (directors of PHMIs and medical education experts) evaluated the 2018 programme and compared results with the 2017 data. The CIPP methodology investigates all outcomes related to education, from the intended positive impacts and short-term results to unintended impacts and long-term results [9]. Therefore, open-ended questions were included in the survey conducted after the completion of the programme to assess unintended outcomes.

Results
In context evaluation, the educational needs of the recipients of the Centre's CPD programme, impediments that inhibit participation in education, and resources that the Centre can utilise were identi ed. Our online survey results were analysed for educational needs assessment ( Table 2). About half of 178 total respondents (49.1%) reported that they chose to participate in educational programmes based on "usefulness to their work." Therefore, it was important to provide practical CPD programmes. When asked what makes it di cult to apply learning to the work environment, 38.4% chose "difference in hospital system," and 36.5% chose "lack of manpower and equipment." This indicates that educational programmes that do not consider the working environments of learners can become obstacles. In addition, 38.1% and 35.3% picked "no one to cover the work while absent" and "inconvenient location," respectively, and 78.5% of the respondents reported preferring 1 day to 1 night 2 days as an appropriate programme period. * The total number of respondents was 204, but 178 responses were analysed excluding missing data. Online surveys with structured questionnaires were conducted and self-administered by respondents.
Similar results were derived using FGIs, which were conducted to achieve an in-depth interpretation of online survey results (Appendix). Many respondents pointed out impediments to participating in the educational programme such as the absence of staff who can cover them at work, and geographical and temporal accessibility. Poor advertisement of educational programmes and a workplace atmosphere that discourages participation in education were also among the responses. These responses clearly indicate that there are many barriers to participation in educational programmes. Instead, increasing the temporal and geographical accessibility of programmes and making education mandatory were suggested as factors that could lead to higher participation. The ultrasound educational programme was the course that respondents most wished to take, because it is the most applicable in primary medical care. However, the ultrasound programme was cancelled several times in 2017 because of lack of enrolment. This shows that the content and delivery of education can prevent learners from enrolling in a programme despite high awareness of its utility, which is problematic.
In consultation with experts, expanding accessibility and providing education content tailored to the roles played by public health institutions in the community were highlighted. In particular, huge gaps were seen in the actual clinical environment and patient groups between the learners and the university hospital where the educational program was outsourced. Based on these results of the online survey, FGIs, and expert consultations, the ability to quickly identify emergent patients or to treat elderly patients with chronic conditions were especially important in the learners' clinical environments. The location of the programme and programme length were also critical.
The abdominal and thoracic ultrasonography programme was determined to require the most changes in needs assessment because of the biggest gap between the willingness to attend the programme and actual participation rate. Moreover, the Centre potentially can be held to account for the cancelled programmes if poorly planned during the inspection of government administration. Therefore, we reviewed records from the 2017 educational programme. Initially, the Centre planned to deliver 3 ultrasonography courses for a total of 10 classes. However, only 6 classes were completed, a potential issue of accountability. Financial support by the Ministry of Health and Education and administrative support by the Centre were secured for the programme.
Through input evaluation, we identi ed the best alternative that satis ed all pre-selected criteria. After consultation with external experts and workshops with stakeholders, responsiveness to priority system needs, potential effectiveness, t with existing services, affordability, and administrative feasibility were selected as the criteria. The abdominal and thoracic ultrasound programme was maintained in 2018 because it met the priority system needs, enabling quick screening of emergency patients. However, it was also clear that the programme required improvement. We selected educational material and providers through open competitive bidding through a targeted outsourcing strategy for the abdominal and thoracic ultrasound course only, a strategy that was determined to be being superior to the outsourcing strategy used in 2017 when a total of 17 CPD programmes were outsourced en bloc to a single contractor. While the Centre provided administrative support, it selected one academic society in a related eld that best ful lled the criteria.
There was one academic society which ful lled the criteria of responsiveness to priority system needs and t with existing services by setting the educational goal as differentiation of emergency diseases and achievement of skills needed for quick referral in primary care. It also met the criteria of potential effectiveness by establishing a practice-centric educational method. By shortening the educational period from 4 nights 5 days in 2017 to 1 night 2 days, we satis ed the criteria of potential effectiveness and t with existing services. The educational programme described in the initial proposal was affordable and administratively feasible. For these reasons, the academic society was selected as the provider of educational content. Also, considering the demand for better geographical accessibility, we decided to provide education not only in Seoul, but also in other areas.
After adopting these changes, the revised abdominal and thoracic ultrasound programme was offered in 2018. It went as planned, and even had to be expanded due to increased demand. The initial goal was to educate a total of 24 clinicians in 2 sessions of 12 students each.
However, as the programme was more accessible than the 2017 programme in terms of time and location, more students signed up than expected. More sessions were scheduled, bringing the programme operation rate to 150% and the education recipient rate to 187%, both being higher than in 2017 (Table 3). Two of the authors observed sessions in real time and recorded progress. The learners, instructors, and operators were asked to complete freeanswer questionnaires. The data obtained were then analysed during a meeting at the Centre, where concerns were raised that necessary capabilities may not have been properly trained due to the shortened education period. However, the responses from learners during sessions indicated that they were not only satis ed, but also quite con dent about their learning. The instructors felt that learners demonstrated the required level of competence. Free-answer questionnaires were collected from all, the instructors, the learners, and the operators. The instructors said that the programme was highly effective, citing intense concentration and enthusiasm among the learners. The learners were satis ed with learning content that could be applied in daily clinical practice and with the diverse case studies they were provided. Many of the learners left comments thanking the instructors and operators. The operators described vibrant interactions between the instructors and the learners.
Goal attainment was examined by looking at post education improvements in the academic achievement (Fig. 1). Academic achievement as measured based on self-e cacy rose from an average of 3.06 out of 10 to an average of 7.9 after the programme. The educational satisfaction levels for 2017 and 2018 programmes (Fig. 2) showed that the 2018 programme scored higher than the 2017 programme in topic satisfaction, teaching method, applicability, and instructor preparedness. In terms of geographic accessibility, the 2018 programme scored lower than the 2017 programme. However, considering that respondents who found the 4 nights and 5 days programme held in Seoul in 2017 too long and too distant did not participate in the rst place and only those for whom it was easy to attend participated, the 2017 participants may have been quite satis ed with the geographic accessibility of the programme location. Based on an analysis of per capita education costs, the 2018 programme cost only 84.1% of the total investment in the 2017 program (Fig. 3).
Three to six months after the completion of the program, the recipients were asked if they experienced progress in their performance by applying what they learned in their work. On a scale of 1 to 10, their answers averaged 7.19. One obstetrician-gynaecologist shared a story in a newsletter about how he successfully improved the diagnosis and survival rates of his emergency patients after receiving ultrasound education. All of short-term and long-term effects were examined and re ected in decision-making. An education and training review committee consisting of o cials from the Ministry of Health and Welfare, public medical institution directors, medical education experts, and physicians conducted evaluations and made decisions as stakeholders. The review committee pointed out that only a small number of students could receive ultrasound education because of the high cost of the programme. Still, the committee agreed that the content of the programme corresponds to the goals of public medical institutions and the Centre's core values, and evaluated the modi cations carried out in 2018 as effective for meeting educational goals. Accordingly, they decided to maintain the program, and the same programme was provided in 2019 as well.

Discussion
In this research, the CIPP model was applied to evaluate CPD programmes targeting physicians employed by public health institutions in Korea. CPD provided to physicians working in the public healthcare sector has great potential to directly improve the quality of community healthcare.
To realise the potential of CPD to the fullest, programme evaluations must be thorough, and improvements to educational programmes must be based on such evaluations. CIPP method makes it possible to highlight the context in which the CPD is provided, which in our case was public healthcare and physicians in the public healthcare sector. In context evaluation, we emphasised the following factors: education is funded by taxpayer money; education should be geographically and temporally accessible to doctors working in diverse environments and facilities; education should rely on administrative support from the Centre. Therefore, during input evaluation we adopted an education strategy that met the criteria of being responsive to priority system needs, potential effectiveness, t with existing service, affordability, and administrative feasibility. Improvements were made accordingly, and better abdominal and thoracic ultrasound courses were provided in 2018. Through process and product evaluation, we demonstrated that the 2018 programme had a higher number of participants compared to the previous year (2017), with higher levels of participant satisfaction and cost effectiveness.
This study has limitations because it analysed only one programme among many provided by the Centre (17 courses in 2017, and 10 courses in 2018) over a short span of time. An educational programme is constantly affected by factors within and outside the program. Therefore, it is clear that a number of programmes provided by the Centre must have mutually affected one another. Needs assessment was carried out through surveys and FGIs, with the subject being the entire educational programme provided by the Centre. The scope of the study was limited to an ultrasound educational programme that was identi ed as high priority through needs assessment. This may have made the analysis of complexity and interrelations between different programmes impossible. Moreover, as FGIs conducted in 2017 were based on voluntary participation, only 10% of the total learners participated in FGIs. Therefore, we cannot dismiss the possibility of bias toward inclusion of enthusiastic learners. Long-term outcomes such as impacts on patient treatment were not evaluated. The study was based on data collected May 2017 to January 2019, when the 2017 programme was initiated to when the applications of practical skills were surveyed for the 2018 program. Therefore, long term outcome or impact of the programme has not been examined fully. Nonetheless, one obstetrician-gynaecologist reported improvements in diagnosis and survival rates of emergency patients after receiving the ultrasound education.
Despite the limitations mentioned above, the study has strengths in being comprehensive, having conducted evaluation at all phases beginning with the previous educational programme, improvement and implementation of the revised programme, and evaluation of the revised programme after implementation. In addition, as ongoing programme evaluation is carried out based on shared understanding, whether yearlong education is being provided as planned can be later reviewed. As we assessed not only whether the intended outcomes were achieved but also how and why they were achieved, the process can be reproduced in the future, and accountability data can be secured in the future as well.

Conclusions
The CIPP approach is suitable for the evaluation and improvement of public health CPD. It enables the analysis of interactions between internal and external factors affecting educational programmes, as well as of the context in which the programme is provided. The CIPP approach thereby helps to produce desired changes in educational programmes and to evaluate and improve programmes funded by taxpayer money, because accountability data is generated while simultaneously executing formative and summative evaluations. CPD programmes administered by the Centre have the potential to improve quality of care provided by physicians in Korean public health and medical institutions, and eventually to enhance public health. The educational programme we examined exhibited marked improvement after the application of the CIPP approach and we were successful in realising our goals.

Consent for publication
Not applicable.

Availability of data and materials
The datasets generated and analysed during the current study are not publicly available due to ethical requirements at the National Medical Centre. However, the educational programs and annual evaluation reports are available from the corresponding author.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. AppendixnationwideCPDevaluation.docx