A multi-agent simulation software, NetLogo, was used for our ABM’s implementation. In the ABM, each agent represents a GP (or a student). Agents are in various states such as STUDENT_1 and TRAINING_1 and have attributes such as age and duration in each states that distinguish themselves from other agents. Over time, an agent may change her/his state from one to another. In our model, each time step corresponds to one year in the real world. The ABM was used to simulate the change of GP workforce in Shanghai, China from 2016 to 2035.
2.1 Conceptual framework
A conceptual framework (see Figure 1) was established mainly based on an extensive policy documents(4, 20, 21) reviews about education, training and employment. According to policy documents “Primary medical and health team building plan focusing on general practitioners” (21) published by National Development and Reform Commission, we can divid the stages in their work course for GPs into college education stage, post-graduation medical education stage, and finally work stage. And we sorted out the behaviors that GPs need to go through and then construct the conceptual framework.
Conceptual framework was abstracted from growth path in GPs’ work course. There are different ways to obtain GP qualification, as presented by GP_1, GP_2, GP_3, and GP_4 in Figure 1. The most mainstream GP education and training is the so-called “5+3” model, that is, commonly it will take five years of undergraduate clinical medicine education and three years of standardized residency training or three years of postgraduate education for a master's degree in clinical medicine for a high school graduate to obtain a residency training certificate (See GP_1). In addition, there are some transitional measures in China to educate and train GPs or recognize the qualifications of GPs. For example, GPs can obtain qualifications by position transition training (general practitioners transferred from other clinical subject) (See GP_4) and "3+2" education and training (three years of college educational of clinical medicine with two years of standardized residency training,finally get college degree of medicine) (See GP_3). In the Shanghai area, one kind of special clinical medical undergraduates is order directed clinical medical undergraduates funded by the Shanghai government. They sign a targeted employment agreement with the Shanghai Municipal Health Commission and the People’s Insurance Bureau, promising to serve in targeted medical and health institutions for six years after graduation; tuition fees will be exempted and living expenses will be subsidized during school (See GP_2). This study focuses on GPs (or students) who have experienced or will experience “5+3” education and training, and restore the evolution of its state, from college education stage, post-graduation medical education stage, and finally work stage. The state evolutions of agents are related with their behaviours. For high school graduates, they first face the choice of whether to enter a medical school or major in clinical medicine. This relates to whether they can enter the medical industry and is the initial threshold for becoming a general practitioner.
If high school graduates are successfully admitted to clinical medicine or general medicine major, they would transite their states into undergraduate medical education as clinical medical undergraduates (STUDENT_1), entering college education stage. And they will be faced with the choice that whether to transfer major or whether to furtherly choose general medicine, which may cause their different working life cycle paths.
After graduation, agents may face with different choices such as employment, graduate study, residency training and so on. Among them, employment involves agents’ whether to employ in the medical industry; residency training involves agents’ whether to choose general medicine and whether to withdraw during the training; graduate study involves agents’ whether to major in general medicine. If clinical medical graduates who meet the graduation requirements become professional master students of general medicine (TRAINING_1) or resident doctors of general medicine (TRAINING_2), they are considered entering post-graduation medical education stage. This stage ends by getting residency training certificate, which means that agents have the qualifications to enter Shanghai to become a general practitioner. After three-year study of postgraduate medical education, agents can get residency training certificate and professional master degree of general medicine. And three-year training of residency training in general medicine is also ended by getting residency training certificate. Agents TRAINING_1 and TRAINING_2 who gets residency training certificate are represented by GP_1.
As for the additional measures, current transitional measures to accelerate the development of GPs, to become GPs, we set up a simple program allowing the transition of its state.
During work stage, GPs are faced with choices like where to work and whether to resign. If agents become GPs in Shanghai, they would switch their states into GP in Shanghai. Some of them are possible to change jobs after working. If they leave the job of GP in Shanghai, they would become RESIGN. The work transfer of GPs in Shanghai is not considered in the model. And agents may retire (RETIRE) when their ages meet the retirements.
There are some "common" losses in above. For example, agents can choose to study or work abroad at any stage; they will die or lose labor during their lifetime. In human resource planning, appropriate consideration should be given to the loss of manpower.
2.2 Parameter and data source
As shown in Table 1, parameters in the model can be divided into three categories: ①Initial data loaded into the model; ②Transformation rate parameters, that is parameters in rate of agent’s one state jump to another; ③Parameters about agent’s duration in one state. The model was calibrated using real data made available by the health or education departments. Parameters were calibrated using official records when available.
Initial data loaded into the model was collected from the reports by health and education authorities. Nationwide residency training in general medicine was standardized in 2014. Taking into account the warm-up period, we set 2016 as the starting year in the model.
Parameters in the rate of agent’s one state jump to another are initially derived from the percentage of agents who jump to another state in agents of the original state, and the value is finally determined after model verification and sensitivity analysis.
In addition, parameters about an agent’s duration in one state came from the requirements of the education or training period in the policy documents formulated by the education and health authorities.
2.3 Model implementation and analysis
Randomnesses were set in the model for the purpose of simplifying the real world suitably and completing the simulation better. For example, students move from STUDENT_1 to TRAINING_1 if there are vacancies. In real life, medical students have to take an exam, and based on the grade obtained they choose from a list of specialties, with the best students having priority over the others. In the simulation model, we do not require such a complex procedure. It is assumed that all medical students have the same probability of becoming master students of general medicine within the vacancy. And if there are more candidates than vacancies those unable to obtain a spot will be considered out of the model. Moreover, agents may drop out at any time during college education and post-graduation medical education stage. For agents who drop out in these two stages, we set them to drop out of GP_1 together, avoiding so many drop-out programs.
In addition, agent-based models are usually stochastic in nature, rarely producing the same result twice even given the same initial parameters. Given this fact, we run the model multiple times to obtain the distribution of the results. After about 80 replications, the median and the range of the major outcome, that is, the size of GP, became stable. Therefore, in this study, we ran 80 replications of each scenario.
As for model verification and validation, we have followed the practices reported in the literature(9, 22-24) for ensuring the model is an accurate representation of the system being simulated and was implemented correctly. To check that the program does what it was planned to do, verification was throughout the whole process. We followed the ABM core design principle and start simple, incrementally verifying the alignment between our conceptual framework and the code(23). While writing code, we often discussed the rules and results of the model with members of the research team, experts in the field of health workforce, and experts in the ABM model to reach a consensus. When it comes to results, we also determined whether the code is actually performing its expected function using graphics presented by NetLogo, and exported the output into Excel, carefully check the results with the values reported by the model. And communication with others for less human error and misunderstanding was throughout the whole process.
To demonstrate whether the simulation is a good model of the target phenomena, validation focus on whether the model can be relied on to reflect the behavior of the phenomena. We used real-world data and designed experiments based on Monte Carlo sampling to test whether the model was behaving as expected(9). Results were matched with historical data, testing whether they reflect the real world. We also compared the results with precious system dynamics model results on Shanghai GPs(11-13). And we elaborated experiments with predictable outcomes to verify if the model behaved according to the expectations(9).
The baseline scenario reflects the evolution of GP workforce in Shanghai until 2035. In sensitivity analysis, the ABM model has been modified with five different transformation rate parameters. The changing parameters have been modified by 10%, 30%, and 50% increasing (+) or decreasing (−), total 39 different scenarios have been modeled to investigate the impacts of parameters on the number of GPs in Shanghai in 2035.
To analyze the influence of changes to policy variables and also to validate the model we have conceived three different scenarios for Rate 3, Rate 4, and Rate 3& Rate 4. Using the literature content analysis method, we tried to find survey on expacted transformation rate for better reflecting the possible scenario compared with our 10% or 30% changes.