Exploration of Evaluation Index for Obstetric Anesthesiologists Training in China Using Modified Delphi Method, Experiences From Pilot Project of PKUHSC


 BackgroundWith the gruadual liberation of fertility policy and the increased number of pregnant women at high-risk in China, a large number of obstetric anesthesiologists (OA) with competency were urgently demanded. In order to establish a replicable and standardized training system for OA specialists, Peking University Health Science Center, one of the first ratified specialists training centers in China, initiated this explorative project. As this is a continuous improving process, the final evaluation index has not been formulated. MethodA working group was set up to devise survey instrument and the initial index draft. We initiated a nation-wide questionnaire pooling opinions of 40 experts from Youth committee of China Anesthesiology Association to construct the evaluation index based on modified Delphi method to reach a consensus. Experts’ demographic information, enthusiasm, authority, and agreement consensus were surgeyed. The final evaluation index would be constructed based on experts’ opinion.ResultsData were collected and analyzed for stability of the experts’ enthusiasm (97.5% and 87.2%, respectively), authority (0.91 ± 0.07 and 0.90 ± 0.13 for the respective two rounds), and agreement consensus (16.7% and 1.3% respectively; Kendall coefficient were 0.26 p = 0.415 and 0.147 p = 0.000, respectively). After two rounds of the survey, the evaluation standard was achieved with 2 items deleted, 5 items added, and 11 major revisions according to experts’ opinion. Revisions mainly focused on clinical practical ability (72%) and the research ability (28%).ConclusionThe final evaluation index was constructed consisting 3 categories (practice ability, non-technical skills, and teaching and research capabilities) with 5 primary and 14 secondary evaluation standards which further divided into 51 items. We hope it could be widely adopted as a standardized evaluation reference for obstetric anesthesiologists training.


Background
Maternal health has been incorporated into the United Nations Millennium Development Goals and the United Nations Sustainable Development Goals [1,2]. Data from United States showed that severe morbidity was highest in women aged > 40 years. [3,4]. With the comprehensive liberalization of fertility policy in China, the number of more older pregnant women and pregnant women at high-risk increased signi cantly [5,6]. A cross-sectional study demonstrated that 81.12% hospitals lacked neuraxial labor analgesia service due to the shortage of anesthesiologists, which contributed to the high percentage of cesarean section rate [7,8]. The the incidence of uterus scarring complicated with placenta previa, uterine rupture, and postpartum hemorrhage was elevated, challenging medical institutions and demanding su cient number of obstetric anesthesiologists (OA) with competency [9,10]. However, the institutions providing obstetrical anesthesia in China are greatly understaffed, and the quality of obstetric anesthesia care from different regions vary signi cantly. There is no subspecialty-trained OAs in China, currently. In order to satisfy the demand for competent OAs and provide high-quality obstetric anesthesia, it is important to establish the subspecialty training system with standardized evaluation index.
In December 2015, the former National Health and Family Planning Commission, the Medical Reform O ce of the State Council and other eight ministries and commissions jointly issued the guidance on the pilot project of establishment of a standardized training system for specialists in China; In June 2017, three pilot specialty training programs were launched and the medical education framework was determined to be "5 + 3 + X". That is, 5 years of medical undergraduate education, 3 years of residential training and 1-4 years of specialty and subspecia;ty training (which equals to fellowship training in the U.S.). It is expected to develop the nationwide medical training system in 2020, and lay foundation for construction of postgraduation medical education system [11].
In order to satisfy urgent needs for OAs, and to establish a replicable and standardized training system for OA specialists, Peking University Health Science Center (PKUHSC), one of the rst rati ed specialists training centers in China, initiated this explorative project. In 2017, PKUHSC formulated the "detailed rules for the current standardized training system of specialized anesthesiologists". The training duration is 3 years. The rst 2 years were "generalized anesthesia training", followed by 1 year of subspecialty training (cardiothoracic anesthesia, pediatric anesthesia, obstetric anesthesia and advanced comprehensive anesthesia). Trainees could select only one subspecialty. OAs training program was launched in 2019 and was in continuous improvements.
PKUHSC has completed the training framework of OAs training project [12]. However, the standardized evaluation index has not been formulated. Therefore, with referencing to the subspecialty training objective/goals (see Supporting Information Appendix S1), we initiated a nation-wide investigation pooling opinions of experts from Youth Committee of China Anesthesiology Association (YCCAA) and formulated evaluation index for OAs using modi ed Delphi method.

Construction of working group
A working group was set up consisting of 8 members (1 director, 5 professors/associate professors, and 2 attending anesthesiologists). The initial survey instrument was devised, including the informed consent form, the instructions, and the questionnaire. The questionnaire consisted of 3 parts. The rst part included surveys on the basic demographics of the experts. The second part surveyed the familiarity and evaluation standards of the experts. The third part is the drafted evaluation index, with three categories which further divided into 48 items (see Supporting Information Appendix S2).

Experts panel selection
We randomly distributed 40 questionnaires to experts (from 40 different hospitals in different regions of China) who should meet the following requirements: members of the YCCAA, with professional titles (professor/associate professor), having teaching experiences, having more than 10 years woking experiences in clinical anesthesia, and any two experts should not work in the same hospital. The process of generating the evaluation index and the overview of the procedure using Delphi method are described in detail in Fig. 1.
The preliminary evaluation index was assessed by the working group for clarity and completeness.
Program directors were briefed on the concept of evaluation index and the purpose of the study prior to the initiation of the study, and was informed regularly with the progress. We agreed that at least 2 rounds of the Delphi surveys should be conducted to achieve a consensus among experts. The decision whether to undertake further rounds of the survey was depended on whether the results from the completed rounds represented a consensus. The questionnaire were distributed to experts by email, collected after prede ned time interval of within one month, and were analyzed independently by two attending anesthesiologists in the working group who were blinded to the participation of the experts.

First-Round of survey
The rst-round of the survey began on June 15th, and was completed on June 30th, 2020. Experts were asked for advice of the preliminary evaluation index, anonymously. In order to quantify the subjectivity of the evaluation, items were rated subsequently on a 4 -point Likert scale (1 = suggesting to delete the item, 2 = major revision of the item, 3 = minor revision of the item, 4 = agree with the item) [1]. Experts were also asked to provide any additional items they felt important and should be included in the evaluation index. Categories, sections and items that were rated by two or more experts as inappropriate, redundant, or incorrect were deleted or revised. In addition, experts were asked to rate the degree of familiarity of the eld.
If only one expert deemed certain items as inappropriate, the working group would discuss and decide whether to delete or reserve these items. Further explanations of these decisions were included in the revised draft and distributed to experts for further comments in the subsequent rounds of the investigation. Additional suggestions made by experts were also incorporated anonymously in the modi ed evaluation index and distributed for further rounds of inquiry.

Second-Round Survey
The second-round survey was performed between July 25th, and August 25th, 2020. We send the revised questionnaire to experts who had completed the rst round. Experts were also provided with the anonymous results from the rst round with the suggestions of revised items. When no further suggestions or signi cant changes were brought forward, the results were compiled to create the nal evaluation index.

Data Analysis
Data were collected and calculated for stability of the consensus, experts' enthusiasm, experts' authority, and experts' agreement [2].

Stability of the consensus
We evaluated the stability of consensus on the basis of the response consistency of the two rounds. In the present study, we considered the stability of consensus to be achieved if at least half of the experts agreed with the items, and there were minimal changes in the selections from the previous round.

Experts' enthusiasm
The experts' enthusiasm of performing the questionnaire were also evaluated. The enthusiasm coe cient (E) is used to evaluate the enthusiasm of experts in participating the research. E=(Ec/Et)/100%. Ec indicates the number of experts who completed the questionnaire. Et is the total number of distributed questionnaire. Expert enthusiasm coe cient is expressed by the recovery rate of the consultation questionnaire, which mainly re ects the degree of experts' attention and involvement in the consultants to the current work. A higher value indicates the degree of attention and interest of the experts in completing the task. The maximum value is 100%.

Experts' authority
The authority of experts (Au) is mainly evaluated by experts' self-evaluation, based on their familiarity with the consulting questions and the degree of judgement strength. Au is the mean value of the experts' academic level (Aac), the familiarity of the experts to the project (Afa), and the foundation of their judgement (Afj). Au =( Aac + Afa + Afj)/3. The weight of experts' academic level were assigned according to their professional and academic ranks and titles (Table 1). Experts' familiarities (Table 2), and evaluation standard were also calculated ( Table 3).

Experts' agreement
Experts' agreement was compared between the two rounds with coe cient of variation (Cv = standard deviation/averaged score). In addition, we calculated the Kendall's co-e cient of concordance (W) by analyzing the quanti ed Likert scale to measure the agreement among experts' opinions [5].

Statistics method
All analyses were performed with SPSS v26 (SPSS Inc., Chicago, IL, USA). Descriptive statistics was performed to summarize the overall data. Kendall's coe cient concordance W was calculated with nonparametric tests.

Demographic information of the experts
39 (97.5%) questionnaires were recovered in the rst round. The demographic information of experts were shown in Table 4. Most of the experts were in 41-50 years group (n = 24, 62%). High academic authority was achieved in these experts, with 14 (36%) associate professors, 25 (64%) professors, and 28 (72%) experts having doctorate degree. 7 (18%) experts were doctoral tutors, and 27 (69%) were post-graduate tutors. All 39 experts worked in grade-A tertiary hospitals with more than 10 years of working experiences.
All experts were from hospitals of residency training base. Among these hospitals, 24 (62%) were standardized training base for anesthesiology specialty, while 21 (54%) hospitals have the intention of launching obstetric anesthesia training program. It is generally accepted that a recovery rate of 50% is the minimum requirement for pooled analysis of expert opinion for stability [13]. Experts enthusiasm usually re ected by experts putting forward their opinions and the recovery rate of the questionnaire. The recovery rates of the two rounds were 97.5% and 87.2%, respectively. Therefore, the stability of the consensus and experts enthusiasm in current survey were satis able.

Experts' authority
The authority of expert consultation is one of the key factors involved in the construction of the evaluation index. Based on previous research, an expert consultation authority coe cient > 0.7 is considered to be reliable. An authority coe cient of 0.8 indicates even higher reliability of the experts' judgement [2,6]. In this study, the authority coe cient of the experts was 0.91 ± 0.07, which indicated that the experts' evaluation perspective are more experienced and reliable.

Experts' agreement
In the rst round, the total number of selected items were "3, 42, 11, 1816" in correspondence to the Likert Scale of "1, 2, 3 and 4", respectively. In the second round, this reduced to 6 (Likert Scale of 3) and 1728 (Likert Scale of 4), respectively. When more than one experts raised different opinions for the same item, this one will be deleted, revised, or replaced. Likert scale was calculated to work out the Cv and the Kendall coe cient. In the rst round, the Cv was 16.7%; while in the second round, the Cv was 1.3%. Cv within 15% is acknowledged as a reliable index for experts agreement [1]. The Kendall coe cient of the two rounds were 0.26 (p = 0.415) and 0.147 (p = 0.000), respectively, indicating consensus have been achieved by the experts in the second round of the inquiry.

Formulation of the evaluation index
In the rst round, major revisions were made in 11 items. 2 items were deleted (4-"anesthesia management of maternal near miss." and 4-"anesthesia management of E0-1 level immediate cesarean section"). 5 items were added (3-"perinatal anesthesia pharmacology"; 3-, "prevention and treatment of anesthesia complications re ux and aspiration, postpartum peripheral nerve injury, etc. "; 4-"transferring of maternal and infants at high risk"; 5-"anesthesia for immediate and emergent cesarean section: highrisk fetal state perinatal, uterine rupture, umbilical cord prolapse, fetal distress, etc."; and 5-"management of acute laryngospasm, bronchospasm and asthma attack"). Major revisions were made in 1 primary evaluation standard and 10 items (  Fast access to clinical information (basic conditions, pregnancy related complications; ASA classi cation, functional status) and make anesthesia plans.
Fetus and newborn assessment (interpretation of fetal heart rate monitoring, APGAR score).

Decision making and implementation of clinical anesthesia
Selection of appropriate anesthesia method and anesthetic drugs according to pathophysiological characteristics of the pregnant women and surgical procedure.
Familiar with the pathophysiological changes to the pregnant women that may be caused by surgical operations.

Anesthesia for high risk pregnant women
Anesthesia for pregnant women with non-obstetric diseases at high-risk : cardiovascular system (congenital heart disease, pulmonary hypertension, aortic aneurysm, cardiomyopathy, heart failure, etc.), respiratory system, endocrine system, hematology, liver and kidney dysfunction, mental illness, nervous system diseases, infectious diseases, multi-drug abuse, severe obesity, septic shock, etc.
Transferring of maternal and infants at high risk.  Ability to coordinate and utilize public health resources, providing necessary medical-related guidance and services.

Professionalism
Possess the professional spirit of assuming professional responsibilities and willingness of following ethical principles.
Ability to continuously improve the quality of clinical anesthesia.

Discussion
Obstetric anesthesia is a subspecialty of anesthesia dedicated to peripartum, perioperative, pain and anesthetic management of women during pregnancy and the puerperium. The unique skills of OAs in emergency resuscitation and critical illness management make them of special importance in perinatal management in high-risk patients [14]. PKUHSC is pioneered in initiating the anesthesia specialty and subspecialty training program in 2019. Its training objectives, duration, specialty curriculums, technical skilles were drafted and approved by Post Graduation Continuing Education Committee of PKUHSC under continuous improvement. In 2021, the draft of the "detailed rules for the current standardized training system of specialized anesthesiologists" was revised to include more speci c regulations. However, the standardized evaluation index was not formulated till now. In the current study, we have drafted the initial evaluation index with reference to OA training objectives, and solicited opinions from YCCAA experts extensively. We hope this evaluation index could be adopted as a reference to help establish standardized assessment tool by other institutions.

Experts selection
In the current study, the Delphi method is adopted as a structured process to collect opinions from experts by questionnaires distributed with controlled opinion feedback [15]. Considering experts from different regions may have di culty in achieving consensus on the nuances, we made an initial draft based on the training objectives of OA subspecialty of PKUHSC. In order to guarantee acquiring comprehensive, reliable and expertise advice, we distributed questionnaires to experts in YCCAA.
Most of the experts were relatively young, holding high academic (72% with doctorate degree) and professional (36% of associate professor and 64% professor) authority in tertiary referral hospitals nationwide, and excelled in clinical practice, medical education and research, and holding concurrent posts as director/deputy director of anesthesiology department, respectively. They witnessed the rapid development of anesthesiology in China in recent decade, and could provide valuable professional insights of medical education in macro-perspective.

Analysis of the revisions in the evaluation index
Analysis of the revised items in the evaluation index revealed that experts' interests were focused on the clinical "practice ability" (including 2 primary evaluation standards which further divided to six secondary evaluation standards and 27 items). Among these, 2 items (4-and 4-) were deleted. 5 new items (3-, 3-, 4-, 5-, 5-) were added, and another 6 items were under major revision (5, 2-, 2-, 3-, 5-,6-). This accounted for 72% (13/18) of all revised ones, and 48% of all "practice ability" items. With the development of ultrasound guided nerve-block technique in obstetric patients and the increased number of cesarean section under general anesthesia, corresponding assessment requirements were added or revised.
No changes were made in the non-technical skills. We speculated that this might be attributed to the di culty of making objective, direct and accurate assessment for trainees concerning the ability of "teamwork, communication skills, and professionalism". The working scenario simulation teaching method and other corresponding teaching strategies were not widely implemented in training hospitals in China. More innovative reforms were to be performed in this eld. It was noticeable that 28% revisions (10-, 12-, 13-, 13-, 13-) were made in the category of "teaching and research capabilities". This accounted for 33% items of the this category. More attentions have been paid to the research ability, and 4 items (44%) were suggested to be revised.

Assessment of research ability
With more emphasis on the research ability for fellowship training worldwide, some advocated extending the training duration exclusively for scienti c research. While other analysis believed that the scienti c output were not increased with proportion to the extended training time, and this might deter the enthusiasm of potential trainees [16]. Teaching and research capabilities, as one of major categories, was also included in our evaluation index. It was intended to have trainees possess basic scienti c research skills, and to lay foundation for them to become clinician-scientists and participate in research programs after years of accumulation in their professional eld. This is also one of the future development goals of research-oriented hospitals.

Future challenges
The implementation of this pilot program is a gradual process. In 2012, the Accreditation Council for Graduate Medical Education (ACGME) o cially recognized and accredited Obstetric Anesthesiology Fellowship. In 2015, the American College of Obstetricians and Gynecologists (ACOG) and the Society for Maternal Fetal Medicine (SMFM) issued the consensus document for improved referral and regionalization of high-risk obstetric services (antepartum through postpartum care), which stipulated that the maternal care centers should be equipped with sta ngs of OBs. Level II (specialty care) need a "board-certi ed anesthesiologist with special training or experience in obstetrics, available for consultation," and for levels III (subspecialty care) and IV (regional perinatal health care centers), a "board-certi ed anesthesiologist with special training or experience in obstetrics is in charge of obstetric anesthesia services." [17].
While in China, there is no such organizations or regulations. During the alternating period of the old and new specialists training system, anesthesilogists nishing "generalized anesthesia training" were permitted in obstetric anesthesia in comprehensive hospitals. Moreover, obstetric anesthesia is a subspecialty which has a higher possibility of being involved in legal suits of adverse neonatal outcomes, and the work load of OA in China were heavy. In addition, the capacity of OA training in PKUHSC teaching hospitals were limited, with only 1-2 positions every year. Considering the training duration, practicing scopes and risks, as well as the supporting incentive policies for career prospects, there was seldom a trainee would select OA as his/her career priority for the moment. More arduous work were to be done to explore how to improve the incentives to attract more trainees to this subspecialty along with establishment and improvement of OA training program. Moreover, maintenance of certi cation to guarantee the quality of competency was not implemented in all specialty and subspecialty in China.
This periodic accreditation for comprehensive evaluation is expected in the future [18]. The enrolled OA trainees were expected to nish the program at the end of the year 2021.

Conclusion
The objective of OA training not only include anesthetic management of parturient at high-risk, but also time -sensitive medical management of critically ill parturients and neonates, and to help graduates become "peri-delivery physicians" [19]. This study used modi ed Delphi method to reach a consensus on the evaluation standard of obstetric anesthesiologists. After two rounds of the survey, the nal evaluation index consisted of 3 categories (practice ability, non-technical skills, and teaching and research capabilities) with 5 primary and 14 secondary evaluation standards which were further divided into 51 items. More attentions were foucused on clinical practice ability and research training. We hope this pilot program could be continuously improved, and help establish a replicable and popularized standardized evaluation index for obstetric anesthesiologists training.

Declarations
Availability of data and material All data generated or analyzed during this study are included in this published article. More detailed information could be found in Supporting Information Appendix S1-S3.