An App Developed for Predicting Nurse Intention to Quit the Job (NIQJ) Using the Artificial Neural Networks(ANN) in Microsoft Excel: Population-Based Questionnaire Study


 Background: Studies in the past have identified factors related to the nursing staff’s intention to leave the unit, institution, and profession. However, none has successfully predicted the nurse's intention to quit the job (NIQJ). Whether NIQJ can be predicted be predicted is an interesting topic in healthcare management. A model to predict the NIQJ for novice nurses in hospitals should be investigated and developed in this mobile computer age. Objective: The aim of this study is to build a model to develop an app for automatic prediction and classification of NIQJ using a smaller number of items to help assess NIQJ and take necessary actions before nurses quit the job.Methods: We recruited 1104 novice nurses working in six medical centers in Taiwan to complete 100-item questionnaires related to NIQJ in October 2018. The k-mean was used to divide nurses into two classes (i.e., NIQJ and Non- NIQJ) based on five- NIQJ items regarding leave intention. Feature variables were chosen from 100 relevant items. Two models, including artificial neural network (ANN) and convolutional neural network (CNN), were compared across four scenarios made up by two training sets (n=1104 and n=804B) and their corresponding testing (n=300a) sets to verify the model accuracy (e.g., sensitivity, specificity, area under the receiver operating characteristic curve, AUC) and stability and generalization (e.g., using the training set to predict the testing set). An app predicting NIQJ was then developed involving the model's estimated parameters as a website assessment.Results: We observed that (1) 24 feature variables extracted from this study in ANN model yielded a higher AUC of 0.82 (95% CI 0.80-0.84) based on the total 1104 cases, (2) the ANN performed better than CNN on both accuracy, stability and generalization, and (3) an ready and available app for predicting NIQJ was successfully developed in this study.Conclusions: The 24-item ANN model with the 53 parameters estimated by the ANN for improving the accuracy of NIQJ has been developed with the use of Excel (Microsoft Corp). The app would help team leader and HR department to pick up nurse’s NIQJ before actions are taken, allowing them to make plans accordingly.


Page 3/17
Background The World Health Organization (WHO) [1] reported that although there is a gradual increase in the number of medical professionals around the world, the gures are far from meeting the rising demands [2] for health care manpower, nursing staff in particular. More than half of nurses with the National Health Service(NHS) have considered quitting the work pressures continue to ratchet up [3,4].
Lack of nursing staff may lead to increased medical incidence because of negligence, involving infection, falls, medication errors, tube dislodgement, pressure sores, and, most seriously, death [5,6]. Nurses overloaded with a high number of patients may experience increased work stress, which has been identi ed as a key factor in increasing nurse turnover [7]. The excessive workload on nursing staff may prolong patient hospitalization, increase patient morbidity and mortality, and increase the incidence of adverse events [8]. Therefore, it is just as important for hospital management to predict Nurses Intention to Quit the Job (NIQJ) as it is to investigate the reason behind the insu cient nursing workforce [9].
Numerous studies [10][11][12][13] have identi ed factors related to a nurse's intention to leave the unit, institution, and profession. However, there has not been any success in predicting NIQJ using the technique of arti cial neural network (ANN) in literatures. The advance computer technique has enabled us to overcome the failure in prediction. Early identi cation of NIQJ may prompt the involvement of supporting service and subsequently alter the outcome, whether it is a longer career for nurses or a buffer period for the management team to make arrangements.

Arti cial Neural Network
The ANN is a component of arti cial intelligence that is meant to simulate a functioning human brain [14]. ANN is the foundation of arti cial intelligence (AI) and solves problems that would otherwise be impossible or very di cult by human statistical standards [15]. It was worthy incorporating ANN, the famous deep learning method, to see if it can improve the prediction accuracy on NIQJ classi cation without having to directly ask the question of quitting [16].

Online Classi cation Using Smartphones is Required
As with the advancements in web-based technologies, mobile health communication is rapidly improving [17]. There was no smartphone app designed to classify NIQJ yet. Once the ANN algorithm learned and estimated the NIQJ model's parameter, the classi cation system can incorporate an early warning response for HR management to react only to those directly relevant to quitting as well as to plan ahead accordingly [16].

Study Aims
The aims of our study are to (1) estimate the model's parameters using ANN based on nurse's responses to questionnaires on both NIQJ and Non-NIQJ and (2) design smartphones based app for assessment on NIQJ.

Study Sample and Demographic Data
If the con dence level and intervals are set at 0.05 and ±5% and applies to the population of 300 novice nurses (about 30% of nurses in hospitals are N0 and N1 in the nursing hierarchy [18]) in a hospital, 169 participants are required to ful ll adequate sample size [19,20]. We estimated the rate of refusal to respond to be around 40%. Therefore, the minimum number of participants for this study will be 282 In October 2018, we delivered 300 copies each to six medical centers in Taiwan, inviting novice nurses(i.e., nurse hierarchy at N0 and N1 only) to complete the 100-item questionnaires (Figure 1 and Additional File1) related to NIQJ [10][11][12][13]. A total of 1,104 nurses participated, with a return rate of 61.3% (Additional File2).
This study was approved and monitored by the NCKU Hospital institutional review board (06476734). All hospital and participants' identi ers were stripped.

Featured Variables
Featured variables were extracted from these 100-item questionnaires using logistic regression with Type error set at 0.05, where the dependent variable(NIQJ as 1 and Non-NIQJ as 0) was determined by k-mean clustering method [21] on the summation scores of NIQJ( Figure 1).

Four scenarios and two models
Model accuracy(e.g.,>0.7) and stability(or, say, generatlization) (e.g. discrepancy between training and testing sets) were focused on out of various facets like model feasibility, e cacy, and e ciency. Firstly, the 1,104 participants were split into training and testing sets in a proportion of 70% to 30%, where the former was used to predict the latter. Four scenarios consisting two training and two testing sets derived from such grouping ratio: Total cases (n=1104) as a training set, its corresponding testing set (n=300), another training sets using 70% of participants (n=804), and its corresponding testing set (n=300). The higher and lower summation scores of NIQJ were used in the training sets, while the middle summation scores of NIQJ were used in the testing sets. Secondly, the accuracy (e.g., sensitivity, speci city, area under the receiver operating characteristic curve, AUC) and stability(or, generalization) ( (e.g., using the training set to predict the testing set) were veri ed. The data is shown in Multimedia Appendix 2.
The ANN and convolutional neural network (CNN) were analyzed with the four scenarios mentioned above. CNN has traditionally been performed on Microsoft (MS) Excel [18,20,22] while ANN has not been paired along with MS Excel in the past. As demonstrated in gure 2 below, the ANN process involves data input in layer 1 where the data joined with two types of parameters and run through the sigmoid function algorithms in layers 2 and 3. Finally, as shown on the right side and bottom of gure 2, the prediction model was deemed complete when the total residuals were minimized through the MS Excel function of sumxmy2 and solver add-in.

Task 1: Comparison of Accuracies on Two Models and Stability across Four Scenarios
The accuracy was determined by observing the higher indicators of sensitivity, speci city, precision, F1 score, accuracy, and AUC in both models. The de nitions are listed below:  A visual representation of the classi cation was plotted using two curves based on the Rasch model [23]. The study owchart and the ANN modeling process are shown in Figure 3 and Multimedia Appendix 3, respectively.

Demographic Data of Participants
The demographic data of the novice nurses (i.e., nurse hierarchy at N0 and N1) are shown in Table 1. Two of the medical centers, B (n=134) and F (n=135), had few participants than the minimal satisfactory sample size of 169. Most participants were aged below 30(1010/1104=91.5%) and worked for privatelyrun hospitals (667/1104=60.4%). Normal distributions were seen in self-assessment of health status and workload. 16.9% identify themselves as being religious. The number of positive NIQJ(=571) is slightly higher than those with Non-NIQJ(=533). Accuracy and Stability as well as generalization in comparison of models When comparing the two models with full data set of 1104 cases, the ANN model scored higher than the CNN model across all six indicators of sensitivity, speci city, precision, F1 score, accuracy, and AUC, suggesting that the ANN model had a higher accuracy.
Thee ANN model also performs better in terms of model stability when comparing the testing results with ACUs (e.g., 0.68>0.59 and 0.78>0.71 in Table 2 and the lower AUC of testing 300 in Figure 4). It is worth mentioning that the group consisting 70% of sample(n=804) has the highest AUC when comparing to other scenarios due to the higher discrimination power caused by selecting, the lower and higher summation scores in NIQJ, as mentioned in the method section.

App Predicting NIQJ for a Web-Based Assessment
The interface of the app targeting novice nurses in order to predict NIQJ was shown on the left-hand side of Figure 5. Readers are invited to click on the links [24,25] and interact with the NIQJ app, see Multimedia Once responses are submitted, it generates a result as a classi cation of either possible NIQJ and Non-NIQJ without having to directly ask the question regarding quitting.
An example is shown on the right-hand side of Figure 5, from which we can see that the participant scored a moderate probability (0.83) of non-NIQJ, which is the curve starting from the top left to the bottom right corner. The sum of probabilities for NIQJ and Non-NIQJ is 1.0. The odds can be calculated with the formula (p/[1-p]=0.17/0.83=0.20), suggesting that this novice nurse has an extremely low probability or tendency to quit.

Principal Findings
We observed that (1)  Previous studies [10][11][12][13] merely identi ed the factors related to the nursing workers' intention to leave the unit, health institution, and profession. Over 476 articles came up with the keyword (nurse intention to leave) [26] on PubMed Central as of October 11, 2020. There has not been a predictive model built for analyzing NIQJ. Although authors developed a Support Vector Machine for predicting nurses' intention to quit using working motivation, job satisfaction, and stress levels as predictors [16], none has demonstrated an online predictive model as we did in this study.
More than half of nurses with the National Health Service(NHS) have considered quitting work pressures continue to ratchet up [3,4]. Although the intention to leave dose not always lead to action(or behavior) [27], predicting nurses' intention to quit is an essential and necessary approach to set up an early warning mechanism in the scope of human resource management [16], considering the constant shortage of nursing staff and the current increasing demands1, 2]. In this study, we veri ed that the ANN could improve the prediction accuracy on NIQJ classi cation, which is novel and innovative, where predictions are made without having to ask the direct question regarding quitting. [16].

Implications and Future Work
The ANN performed better than the CNN in both accuracy and stability. In this study, the sensitivity and speci city have been improved. We have not seen others using the ANN approach to predict NIQJ in the literature, which is a breakthrough in this study. We have also not noticed any article incorporating accuracy and stability as well as a generalization to verify model feasibility, e cacy, and e ciency, but many authors have used the split scheme of 70:30 ratio invalidating their predictive models [18.20,28] Over 2,062 articles have been found searching the keyword (arti cial neural network)[Title] on PubMed Central on October 10, 2020. None of them used Microsoft Excel to perform the ANN. The interpretations for the ANN concept and process, as well as the parameter estimations, are shown in Figure 2, Multimedia Appendix 3, and in the app [24,25]. Readers can adjust the parameters in the ANN model on their own and examine the difference in results.
As the quality control process emphasized the principle to consider more with the vital few and less with the trivial numerous [20], we suggest adapting with the matching personal response scheme for correct classi cations in the model (MPRSA) and further increase the accuracy toward 100% [20] in the ANN model. Because the reason being that the same response string will be matched by the MPRSA and lead to a correct classi cation if the responses are identical to that of the original dataset. We recommend searching individual responses in the dataset rst. If found, assign the correct classi cation to this respondent. Otherwise, the classi cation will be determined by the NIQJ ANN model; see Multimedia Appendix 5.
Furthermore, the curves of category probabilities based on the Rasch model [23] are shown in Figure 5. The binary categories (e.g., success, and failure on an assessment in the psychometric eld) have been applied in health-related outcomes [29][30][31][32][33]. However, we are the rst to provide the NIQJ animation-type dashboard on Google Maps, as shown in Figure 5.

Strengths of this study
ANN was performed on Microsoft Excel, which is rare in the literature. An app was designed to display classi cation results using the category probability theory in the Rasch model. The animation-featured dashboard was incorporated with the ANN model and allow an easy understanding of the classi cation with visual representations.

Limitations and Suggestions
Our study has some limitations. First, although the psychometric properties of the 24-item NIQJ assessment have been validated, there is no evidence to support that it is suitable for novice nurses in other regions. We recommend additional studies using the same approach with the ANN or other models to estimate the parameters and explore the difference and similarity to this study.
Second, we have not discussed possible further improvement in predictive accuracy. For instance, whether other featured variables (e.g., variables are not included in Figure 1) applied to the ANN model can increase the accuracy rate is worth discussion. It would be useful in the future to look for other variables that can improve the power of the model prediction.
Third, the study was carried out on the ANN model whether other predictive models have higher accuracy and stability than the ANN has yet to be investigated.
Finally, the study sample was taken from novice nurses in Taiwan. The model parameters estimated for the NIQJ are only suitable for the Chinese (particularly Taiwanese) health care settings. Generalizing these NIQJ assessment ndings (e.g., the model parameters) might be di cult and constrained because the sample only took into consideration novice nurses working for inpatients. Additional studies are required to re-examine whether the psychometric properties of the NIQJ assessment are similar for inpatients and other sites..

Conclusion
We demonstrated in this study: (1)  This study was approved and monitored by the NCKU Hospital institutional review board (06476734). All hospital and participants' identi ers were stripped.

Consent to publish
Not applicable.

Availability of data and materials
All data used in this study are available in Additional File les.

Competing interests
The authors declare that they have no competing interests.

Funding
There are no sources of funding to be declared.

Authors' Contributions
TH conceived and designed the study. YT performed the statistical analyses and was in charge of recruiting study participants. TWC helped design the study, collected information and interpreted data. HFL monitored the research. All authors read and approved the nal article.