Development and Validation of the Perceived Stress Scale in Emergency Medical Teams During the Epidemic of COVID-19

doi:10.21203/rs.3.rs-665920/v1

Download PDF

Research

Development and Validation of the Perceived Stress Scale in Emergency Medical Teams During the Epidemic of COVID-19

https://doi.org/10.21203/rs.3.rs-665920/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

During the epidemic of COVID-19 of China, the emergency medical teams are facing serious stress in the front-line. As far as we know, there are no studies to test the applicability and measurement properties of the 10-item Chinese perceived stress scale (CPSS-10) in the emergency medical team.

Methods

From March 17 to 27, 2020, an online survey was conducted on the emergency medical teams of Liaoning Province who supporting Wuhan. The CPSS-10 was used to measure the stress of medical workers. Classical test theory (CTT), bifactor model and multidimensional graded response model (MGRM) were used to analyze the measurement characteristics and differential item functioning (DIF) of CPSS-10.

Results

The Cronbach's alpha coefficient of CPSS-10 was 0.86. Bifactor model confirmed that CPSS-10 was a two-factor structure. MGRM showed ordered response categories of K10. Item 8 could distinguish individual stress, but the slope of this item was very large (slope is 7.97, which was higher than 4), showing local dependence. There was a significant age DIF, but no DIF in gender. After removing the items 2, 5, and 8, the CPSS-7 showed high reliability, without DIF of age and gender, and there was no local dependence.

Conclusions

MGRM could provide useful measurement information about CPSS-10 and CPSS-7. MGRM found that CPSS-10 did not fully conform to the item response theory (IRT). CPSS-7 had proved to be a more effective and reliable tool for assessing the perceived stress of emergency medical team.

Health Economics & Outcomes Research

Health Policy

Perceived stress scale

Emergency medical team

COVID-19

Bifactor model

Multidimensional graded response model

The twenty-first century witnessed the challenges of infectious diseases, and new or re-emerging infectious pathogens which remained the major causes of morbidity and mortality by infectious diseases [1]. COVID-19 had attracted attention after the report of unexplained pneumonia in Wuhan, China [2, 3]. It was caused by SRAS-COV-2 virus infection, and subsequently spread to many other parts of the world through global travels [4]. In January 2020, the WHO designated COVID-19 as a public health emergency [5]. At present, COVID-19 had developed into a global pandemic [6]. According to incomplete global statistics before 10 January 2021, there had been 88.38 million confirmed cases of COVID-19, including 1.92 million deaths, reported by WHO [7]. The wide spread of COVID-19 might monopolize government activities and even cause fear and hysteria [8].

During the COVID-19 epidemic, people all over the world are suffering from fear, anxiety, and panic, especially the health care workers, who are facing more huge emergency medical pressure [9]. Health care workers have long shifts, lack of personal protective equipment, fear of bring infection to their families, and face high risk of infection with COVID-19[10], which aggravate their perceived stress. Previous studies have found that the COVID-19 pandemic has a serious impact on the mental health of health care workers and the general population [11–14]. According to the research on the outbreak of severe acute respiratory syndrome and Middle East respiratory syndrome (MERS), anxiety and fear are the first symptoms of health care workers, but the emergence of depression and post-traumatic stress symptoms will cause serious consequences and have a long-term impact on the mental health of health care workers [15, 16].

Medical emergency rescue was an important resource to deal with public health emergencies. By February 23, 2020, more than 330 emergency medical teams and 41600 health care workers had supported Hubei Province [17]. Emergency medical teams were at high risk of being infected with SRAS-COV-2 during their participation in the treatment of COVID-19 patients. As of February 11, 2020, a total of 1716 health care workers were confirmed to be infected with COVID-19 in mainland China, accounting for 3.8% of all confirmed cases [18]. Recently, a meta-analysis found that there was a high prevalence of anxiety, depression and insomnia for the health care workers who participated in the diagnosis and treatment of patients infected with COVID-19 [19]. The higher the frequency of contact with COVID-19 patients, the higher the risk of mental disorders [20]. Previous studies had found that health care workers in emergency medical teams had higher psychological pressure [21, 22]. Therefore, it was necessary to study the mental health of medical workers in emergency medical teams by means of stress measurement scales.

One of the most popular stress measurement scales [23], the perceived stress scale (PSS) had been used to test the mental health of people in different countries and races during the epidemic of COVID-19 [24, 25]. Although PSS-10 was widely used in the studies about mental health of emergency medical team, few concerned about the applicability and measurement properties of it, and the verification research of measurement properties of the 10-item Chinese perceived stress scale (CPSS-10) in the emergency medical team was even less. Meanwhile, most psychological assessments tended to measure two or more latent trait or multidimensional, and CPSS-10 was a Likert scale, its items were graded hierarchically. In order to overcome the limitations of ranked data and multidimension of CPSS-10, multidimensional graded response model (MGRM) should be used for items analysis [26]. Therefore, based on the mental health survey data of front-line medical staff in Hubei Province, the classical test theory (CTT), bifactor model and MGRM were used to evaluate the psychometric properties of CPSS-10 in this study, which could provide the reference for the appropriate revision of the scale. The accuracy of PSS score could help the government understand the psychological pressure of the front-line medical staff more effective.

Study design and participants

From March 17 to 27, 2020, an online survey was conducted among the medical workers of emergency rescue medical team, including doctors, nurses, and team leaders of Liaoning Province who were chosen to alleviate pressure on Wuhan in Hubei province during the epidemic of COVID-19. The study team communicated with the medical rescue team leaders of Liaoning Province many times and made a detailed plan about the distribution and collection of the questionnaire. Finally, we decided to develop an electronic questionnaire and sent it to the emergency rescue medical team through the Web-based survey tool Sojump (https://www.wjx.cn/ accessed on 13 March 2020). We trained the medical team leaders online through Wechat (Tencent Corp). And the trained team leaders guided the team to finish the electronic questionnaire with good quality. The data were collected in excel format through platform of Sojump.

A total of 352 emergency medical team workers (300 females) were in study. All the health care workers who filled in the electronic questionnaire signed the informed consent forms.

Measures And Instrument

The 10-item perceived stress scale (PSS-10), a popular self-report tool, represents the perceived stress of participants in the past month [23]. PSS-10 includes two factors. Six negative wording items (items 1, 2, 3, 6, 9 and 10) constitute the first factor, and four positive wording items (items 4, 5, 7 and 8) constitute the second factor [27]. Specifically, the first factor represents the perceived helplessness subscale (PHS), and the second factor reflects the perceived self-efficacy subscale (PSES) [28]. Items 1, 2, 3, 6, 9 and 10 score from 0 to 4; items 4, 5, 7 and 8 score from 4 to 0. The PSS score lies between 0 and 40 [23]. In this study, the CPSS-10 [29] was used to measure the perceived stress of front-line medical staff in emergency medical teams.

Factor Variables Of Differential Item Functioning

Age and gender are the most important factor variables. In our study, age was transformed into binary variable. So, participants could be classified as adults (aged 22–34 years old) and middle-aged (aged 35–59 years old) according to age.

Statistical Analysis

Bartlett sphericity test [30] and Kaiser Meyer Olkin (KMO) test [31] were performed to offer empirical support for the adequacy of the research data. The null hypothesis of Bartlett's sphericity test showed that there was no significant difference between the correlation matrix and the identity matrix, which meant that the variables were uncorrelated and unsuitable for structural detection [30]. In the actual analysis, when KMO value was more than 0.8, which shows that the effect of factor analysis was better [31]. The Cronbach’s alpha coefficient was used to test the scale reliability.

To determine the appropriate factor structure of CPSS-10, an exploratory factor analysis (EFA) was conducted on the samples of emergency medical team by maximum likelihood estimation and oblimin with Kaiser normalization [32]. Confirmatory factor analysis [33] was further used to test the dimensions of CPSS-10 and verify the applicability of the two-factor and one-factor models [34]. Based on the factor structure reported in PSS literature, our study sets up three different models of CPSS-10: (a) One-factor model [34], assuming that all 10 items measure a single stress factor. (b) Two-factor model [29], which is set as a two-factor structure. (c) Bifactor model [35], which consists of six negative projects and four positive projects. The bifactor model allows researchers to test whether CPSS-10 has other general measures of perceived stress. See Fig. 1 for details.

As the one-dimensional verification index used in bifactor model, we calculate the explained common variance (ECV) and percentage of uncontaminated relations (PUC). ECV is the ratio of common variances attributed to general factors [36]. Higher ECV value indicates that compared with other specific group factors, the data has a strong general factor. When the ECV value is greater than 0.70, the scale can be regarded as one-dimensional, otherwise it supports bifactor model. PUC reflects the degree of item correlation only affected by global factors in the bifactor model [36]. When PUC > 0.8, a group of two-factor model data fitting single dimensional model can get unbiased estimation, that is, the scale conforms to the one-dimensional model.

In this study a set of statistical criteria determine were used to determine the goodness [37], including: Chi-square/df (≤ 3), a comparative fit index (CFI, ≥ 0.90), a tucker Lewis index (TLI, ≥ 0.90), a root mean square error of approximation (RMSEA, ≤ 0.08), and a standardized root mean square residual (SRMR, ≤ 0.08). When the goodness of fit index meets the above conditions, it means that the model is well fitted.

For further evaluating the item fitting of CPSS-10, MGRM was also used to analyze the items. MGRM is a probability model, which describes the response of the participant to any given item according to the level of the participant’ latent trait [26]. DIF is an indispensable part of psychometric analyses aiming to measure invariance across sample groups, e.g., for male and female. The DIF of this study was tested by MGRM [26].

The CPSS-10 scale was revised according to the following strategies. First, according to DIF, if DIF is statistically significant (P < 0.005) after correction, it was deleted. Secondly, if the discrimination estimation of an item was larger than 3.00, it was deleted [38] according to edge maximum likelihood estimation. Thirdly, according to Wright and Linacre's suggestion: when the information-weighted fit statistic (infit) mean square (MNSQ) and outlier-sensitive fit statistic (outfit) MNSQ values of an item was greater than 1.3 or less than 0.7, the fitting effect of the item was poor [39]. Finally, the performance of CPSS-10 and the revised version was evaluated based on the goodness of fit.

All statistical analysis and graphic plotting were performed using R version 4.0.3 R software (The R Foundation for Statistical Computing, Vienna, Austria). And the software packages “mirt” and “ltm” were used to build the MGRM and bifactor model.

Exploratory factor analysis (EFA)

The KMO test value of CPSS-10 in this study was 0.88, and Bartlett test of sphericity was significant (Chi-square = 1703.26,P < 0.01), which met the prerequisite for EFA. EFA results showed that eigenvalues of the first two components were greater than 1 (4.62 and 1.91), so the two factors structure is suitable. Factor 1 was composed of six negative items (called PHS subscale), accounting for 46.15% of the variance. Factor 2 was composed of four positive items (PSES subscale), explaining 19.13% of the variance (Table 1). There was no double loading in the pattern matrix, and the loading of all valid items was greater than 0.5. In addition, there was a correlation between PHS and PSES (r = 0.39, P < 0.05).

Table 1

Exploratory factor analysis (EFA) and reliability of CPSS-10
PSS-10 items (In the last month, how often have you…)		Factor loadings
PSS-10 items (In the last month, how often have you…)		PHS	PSES
Item 1	…been upset because of something that happened unexpectedly?	0.81	-0.06
Item 2	…felt that you were unable to control the important things in your life?	0.85	-0.06
Item 3	…felt nervous and ‘stressed’?	0.81	-0.03
Item 4	…felt confident about your ability to handle your personal problems?	0.01	0.72
Item 5	…felt that things were going your way?	-0.08	0.68
Item 6	…found that you could not cope with all the things that you had to do?	0.75	0.05
Item 7	…been able to control irritations in your life?	0.02	0.55
Item 8	…felt that you were on top of things?	0.11	0.79
Item 9	…been angered because of things that were outside of your control?	0.75	0.04
Item 10	…felt difficulties were piling up so high that you could not overcome them?	0.71	0.11
Eigenvalue		4.62	1.91
% of variance explained		46.15	19.13
Total% of variance explained		65.28
Cronbach’s alpha coefficient of Factor 1 and 2		0.91	0.77
Cronbach’s alpha coefficient of the whole scale		0.86
Inter-factor Pearson’s correlation (2-tailed)		0.39
Note. PSS-10 10-item Perceived Stress Scale; PHS Perceived helplessness Subscale (First common factor); PSES Perceived Self-Efficacy Subscale (Second common factor).

Confirmatory Factor Analysis

Table 2 showed the goodness of fit of the CFA model for CPSS-10. The one-factor CFA model had poor fit, Chi-square/df = 12.02, CFI = 0.77, TLI = 0.71, RMSEA = 0.18 (95%CI [0.12–0.15]), SRMR = 0.13. All indices of fit goodness did not meet the statistical criteria. However, the two factor of CFA model showed a good fit. The fitting index, Chi-square/df = 2.95, CFI = 0.96, TLI = 0.95, RMSEA = 0.07(95%CI [0.06,0.09]), SRMR = 0.04, all of them met criteria well. The results of model 3 in Table 2 showed that PSS-10 of bifactor model was preferred with fitting effect, Chi-square/df = 1.62, CFI = 0.99, TLI = 0.99, RMSEA = 0.07 (95%CI [0.06,0.09]), SRMR = 0.07, which was better than that of two-factor of CFA model (Chi-square = 73.385, P < 0.001). Figure 1 showed the schematic representations of one factor model (a), two factor model (b) and bifactor model (c) of CPSS-10.

Table 2

Comparison between confirmatory factor analysis and bifactor model
Model	Chi-square	df	Chi-square/df	CFI	TLI	RMSEA (95%CI)	SRMR
Model1	420.55	35	12.02	0.77	0.71	0.18(0.16,0.19)	0.13
Model2	100.46	34	2.95	0.96	0.95	0.07(0.06,0.09)	0.04
Model3	40.44	25	1.62	0.99	0.99	0.04(0.01,0.06)	0.07
Note. CFI Comparative fit index; SRMR standardized root-mean-square residual; RMSEA room-mean-square error of approximation; CI confidence interval; df degree of freedom; TLI tucker Lewis index.
Model1: one-factor of CFA model
Model2: two-factor of CFA model
Model3: PSS-10 of the bifactor model

Unidimensionality

Principal component analysis (PCA) showed that the residual explained 46.77% of the original variance (less than 50%). The ratio of the first factor eigenvalue to the second factor eigenvalue is 2.58 (e.g., < 3). Two factor structure of CPSS-10 was found in both EFA and CFA, as shown in Table 2. In addition, Martin-Loef-Test found that LR value = 365.186, P < 0.001, which indicated that CPSS-10 may not be an unidimensional scale. The bifactor model showed that PCU = 0.53 < 0.8, and ECV = 0.65 (which was less than 0.70), indicating there were not only general factor, but also specific factors. These above results confirmed the multidimensional of CPSS-10 from different aspects.

Reliability

In our study, the overall, PHS subscale, and PSES subscales Cronbach's alpha coefficient were 0.86, 0.91 and 0.77, respectively (Table 1). The overall reliability and PHS subscales were high, while the reliability of PSES subscale was only acceptable. After removing item 5 and item 7, the Cronbach's alpha coefficient increased, which were 0.863 (95%CI [0.833,0.886]) and 0.865 (95%CI [0.836,0.889]) respectively, while deleting any other item would reduce the Cronbach's alpha coefficient. This result indicated that the CPSS-10 could be further optimized (Table 3).

Table 3

Cronbach’s alpha and goodness of PSS-10
Item	Outfit	Outfit.z	Infit	Infit.z	Cronbach’s alpha after removing the item (95%CI)
Item 1	0.838	-1.615	0.856	-1.677	0.840 (0.807,0.867)
Item 2	0.717	-2.405	0.807	-2.274	0.838 (0.800,0.868)
Item 3	0.778	-1.781	0.850	-1.726	0.838 (0.803,0.867)
Item 4	1.068	0.644	0.991	-0.046	0.856 (0.827,0.881)
Item 5	1.038	0.420	0.952	-0.482	0.863 (0.833,0.886)
Item 6	0.810	-2.074	0.853	-1.764	0.839 (0.807,0.868)
Item 7	1.204	1.784	1.077	0.768	0.865 (0.836,0.889)
Item 8	0.064	-3.318	0.220	-6.569	0.845 (0.815,0.872)
Item 9	0.807	-2.100	0.860	-1.623	0.838 (0.804,0.866)
Item 10	0.819	-1.585	0.903	-1.089	0.837 (0.802,0.865)
Note. PSS-10 10-item Perceived Stress Scale; CI confidence interval; Infit Information-weighted fit statistic; Outfit outlier-sensitive fit statistic.

Multidimensional Graded Response Model (Mgrm) Analysis

Infit and outfit MNSQ values were used to evaluate the fitting of items. MNSQ of infit and outfit for MGRM were 0.807–1.077 and 0.717–1.204 respectively, which showed an overall good fitting effect of CPSS-10. However, the fitting effect of item 8 was poor with infit and outfit MNSQ value 0.064 and 0.220, respectively (Table 3).

CFA combined with MGRM analysis was used to test the CPSS-10 structure of the emergency medical team. The loads of all items were greater than 0.60, and no item showed disorder threshold (Table 4 and Fig. 2). Items 1, 2, 3, 6, 9, and 10 had higher loads on the coefficient λ₁ (PHS subscale), while items 4, 5, 7, and 8 had higher loads on the coefficient λ₂ (PSES subscale), which confirmed the stability of two-factor structure CPSS-10. MGRM showed that the correlation between PHS and PSES was 0.535. In addition, category probability curves of items 5 and 6 were provided, see Fig. 2A and 2B, which showed that the items could distinguish personnel ability and project difficulty. Other items of CPSS-10 showed similar category probability curves (Additional file 1: Figure S1).

Table 4

Confirmatory factor analysis combined with MGRM for 2-factor structure of CPSS-10
Item	\({\lambda }_{1}\left(se\right)\)	\({\lambda }_{2}\left(se\right)\)	\({a}_{1}\left(se\right)\)	\({a}_{2}\left(se\right)\)	\({b}_{1}\left(se\right)\)	\({b}_{2}\left(se\right)\)	\({b}_{3}\left(se\right)\)	\({b}_{4}\left(se\right)\)
Item 1	0.79(0.06)		2.81(0.27)		-1.14(0.21)	-0.23(0.03)	1.45(0.31)	2.51(0.66)
Item 2	0.82(0.06)		3.23(0.32)		-0.81(0.10)	0.29(0.11)	1.75(0.48)	2.60(0.91)
Item 3	0.80(0.07)		2.94(0.28)		-0.77(0.16)	0.08(0.03)	1.60(0.34)	2.30(0.57)
Item 6	0.76(0.05)		2.59(0.24)		-0.97(0.18)	0.44(0.09)	2.19(0.24)	3.21(1.01)
Item 9	0.77(0.06)		2.62(0.22)		-1.09(0.25)	0.21(0.05)	1.87(0.29)	2.75(0.68)
Item 10	0.75(0.06)		2.54(0.24)		-0.62(0.11)	0.70(0.14)	2.20(0.44)	3.03(0.82)
Item 4		0.70(0.07)		2.06(0.18)	-0.98(0.21)	0.72(0.17)	1.95(0.29)	2.43(0.39)
Item 5		0.62(0.08)		1.62(0.17)	-2.10 (0.28)	0.27(0.14)	1.92 (0.23)	2.92(1.01)
Item 7		0.86(0.09)		1.72(0.17)	-0.88(0.17)	0.90(0.17)	2.04(0.26)	2.73(0.38)
Item 8		0.58(0.08)		7.97(0.67)	-0.95(0.63)	0.80(0.50)	1.76(1.35)	2.16(1.50)
Note: λ indicate factor loadings; α is the item discrimination (slope) parameter; b₁to b₄ are item response category difficulty (threshold, location) parameters; Bolded items indicate the items chose for short form of CPSS-10; se standard error.

Table 4 summarized the slope (α, discrimination) and difficulty (b1 to b4, threshold or location) parameters of 10 items in MGRM analysis. Item 8 and item 2 had high slopes (7.97 and 3.23), while item 1, item 3, item 6, item 9, and item 10 had medium slopes (2.54–2.94). Item 8 had the largest slope and item 2 had the second largest slope. Although item 8 was the most effective way to distinguish individual stress on CPSS-10, it had a very high slope compared with other items (e.g., > 4), and the results of MGRM showed a local dependence.

Expected Score And Item Information Function Of Cpss-10

As shown in Fig. 2C and 2D (e.g., items 5 and 6), the expected score (cumulative probability of category probability curves) increased with the increase of latent trait (\(\theta\)) of emergency medical team members. The item information function (IIF) of items 5 and 6 of CPSS-10 were shown in Fig. 2E and 2F. Generally, IIF reached the maximum when the potential trait (\(\theta\)) was between − 2 and 3. When the latent trait (\(\theta\)) was close to -3 or 3, the IIF was the smallest (close to zero). This result indicated that CPSS-10 showed better discrimination ability among emergency medical workers with medium ability level, rather than those with the lowest or highest ability level. Other expected score and item information function of CPSS-10 were showed in Supplementary information (Additional file 1: Figure S2 and S3).

Differential Item Functioning (Dif)

The DIF of CPSS-10 was determined by gender (male/female) and age (Younger:22–34; middle-age: 35–59). We used likelihood ratio Chi-square to test DIF. MGRM analysis found that DIF of item 2, 5, and 8 were all statistically significant across age groups. However, the DIF of item 2 and 5 were not statistically significant when Bonferroni adjustment was taken with α value (0.05/10 = 0.005). However, DIF of item 5 was still statistically significant among age groups. No significant DIF was found in gender (Table 5).

Table 5

DIF on age (adults and middle-aged) and gender (male and female) in CPSS-10 and CPSS-7
Item	DIF on age				DIF on gender
	CPSS-10		CPSS-7		CPSS-10		CPSS-7
	Wald\({\chi }^{2}\)	p	Wald\({\chi }^{2}\)	p	Wald\({\chi }^{2}\)	p	Wald\({\chi }^{2}\)	p
PHS
Item 1	0.571	0.450	1.206	0.272	0.098	0.755	0.054	0.816
Item 2	5.365	0.021			0.168	0.682
Item 3	2.689	0.101	3.344	0.067	0.599	0.439	0.835	0.361
Item 6	0.023	0.879	0.099	0.519	0.230	0.632	0.659	0.417
Item 9	2.809	0.094	1.267	0.260	1.210	0.271	0.694	0.405
Item 10	0.001	0.987	0.004	0.951	0.006	0.938	0.289	0.591
PSES
Item 4	0.622	0.430	0.416	0.519	0.351	0.553	0.103	0.748
Item 5	10.365	0.001			0.618	0.432
Item 7	0.602	0.438	0.066	0.260	1.515	0.218	0.765	0.382
Item 8	3.948	0.047			0.162	0.687
Note: PHS Perceived helplessness Subscale; PSES Perceived Self-Efficacy Subscale; DIF Differential item functioning; Adults aged 22–34 years old; Middle-aged aged 35–59 years old.

Revision Of Cpss-10 Scale

Item 5 was deleted according to DIF (corrected P < 0.005). Then items 2 and 8 were deleted for their discrimination beyond the interval [− 3.00, 3.00]. According to infit and outfit MNSQ values, the items with poor fitting were deleted. Finally, the CPSS-7 Version (composed of 7 items) was obtained. Cronbach's alpha coefficient of CPSS-7 was 0.82, and CPSS-7 was highly correlated with CPSS-10 (r = 0.98, P < 0.001). It was found that CPSS-7 was more effective in identifying stress perception than CPSS-10. In addition, no DIF difference was found in gender and age of CPSS-7 (Table 5).

In this study, we aimed to evaluate the reliability and validity of CPSS-10, test the psychometric properties of CPSS-10 in medical staff of emergency medical team during the COVID-19 epidemic. In the previous studies, CPSS-10 was detected to have a high level of internal consistency and reliability [29]. CTT and IRT provided sufficient evidence to confirm the multidimensional and two factor structure of CPSS-10 [35, 40]. In our study, PCA, bifactor model and MGRM all supported the two-dimensional structure of CPSS-10. MGRM analysis showed that the response categories were in order of severity. Although there was a significant DIF in age of item 5, there was no significant DIF in gender and age of other items of CPSS-10. In addition, this study had enough psychometric evidence to prove that the revised version of CPSS-7 was reliable and effective.

The Cronbach's alpha coefficient of CPSS-10, PHS and PSES subscales were 0.86, 0.91 and 0.77, respectively. This indicated that CPSS-10 had good internal consistency reliability, which is in line with the results of other PSS-10 studies (the reliability versions range from 0.78 to 0.91) [29, 41, 42]. In addition, a positive correlation was found between PHS and PSES subscale of CPSS-10, this result was consistent with other validation studies in China [43] and South Korea [44], but contrary to the results in Greece [45].

Our study showed that there might be a bifactor structure of CPSS-10. The results of EFA, CFA and bifactor model indicated that bifactor model was best. The bifactor model of CPSS-10 had a general stress factor and a two specific factors structure [28, 46]. Six negative items were loaded into the perceived helplessness factor, and the other four positive items were loaded into the perceived self-efficacy factor. Our result was accordant with Wiriyakijja’ view [46]. However, our study was not only different from the two-factor solution proposed by Pereira et al [47], but also different from the bifactor structure of five negative items and five positive items, which was put forward by Park et al [44].

IRT models, such as MGRM, are not common in health studies. Typically, the measurement characteristics and dimensions of a scale are evaluated by CTT and factor analysis [48]. IRT analysis results in our study showed that the items of PHS and PSES were well fitted by MGRM. The parameterization of items showed that both PHS and PSE can effectively measure and distinguish the potential perceived stress level of grade data [26]. Our study demonstrated an ordered threshold in the category probability curves, which meant that medical workers with more stress on the item did have more anxiety than those who claimed to have less trouble. In other words, CPSS-10 could accurately distinguish low-level and high-level items [49]. However, the discrimination between item 2 and item 8 in cpss-10 was beyond the recommended range, and item 8 showed strong local dependence (e.g., > 4) [50]. However, some previous studies did not find the local dependence of PSS-10 [44, 51].

This study found that items 2, 5, and 8 showed significant DIF in age (P < 0.05). In addition, the items of CPSS-10 did not show DIF in gender. However, an American study [52] showed that PSS-10 had no significant DIF both in gender and age. Based on the findings of DIF in gender and age, it was necessary to verify and revise CPSS-10.

After deleting some items from the scale, some redundancy can be eliminated, and the performance of the scale may be improved [53]. A previous study showed that when K10 (0.93) and K6 (0.89) were compared, some redundancy appeared in Cronbach's alpha coefficient [54]. Our analysis confirmed that the Cronbach's alpha coefficient (0.83) of CPSS-7 was almost equivalent to that of CPSS-10 (0.86). In addition, CFI, TLI and SRMR of CPSS-7 and CPSS-10 were 1.00, 1.01, 0.06 and 0.99, 0.99, 0.07, respectively. All of these indicators showed goodness of CPSS-7. Therefore, CPSS-7 could replace CPSS-10 to measure the perceived stress of emergency medical team. In addition, our study found that the revised CPSS-7 did not show DIF both in gender and age, which meant that there was no gender or age bias in the CPSS-7 [26]. Therefore, the CPSS-7 developed and validated by MGRM is an effective measurement tool with good discrimination ability and high item information function, which can be used to quantify the non-specific psychological severity of emergency medical team members during the epidemic of COVID-19.

There are some limitations in this study. Firstly, during the period of COVID-19 epidemic, online survey was adopted because it was impossible to conduct face-to-face survey with the respondents. Although quality control was emphasized, the difference between online survey and traditional field survey was not considered. Secondly, all diagnoses in this study were mental health symptoms/states, not mental disorders. Thirdly, we could not calculate the response rate due to the participants’ anonymously recruiting. Finally, our study did not investigate the pre-existing stress and anxiety of medical team members. Some respondents may have these baseline stresses and anxieties, so the reported scores may not all be attributed to the COVID-19.

CTT, bifactor model and MIRT showed that CPSS-10 had high reliability and effectiveness in emergency medical team during the epidemic of COVID-19. However, the study recognizes that due to cultural differences and strict adherence to MIRT characteristics, the CPSS-10 needed to be modified to measure stress in Chinese medical workers. The CPSS-7 provided a more powerful measurement performance than the CPSS-10. CPSS-7 met most of the hypotheses of MGRM, which did not show DIF both in gender and age, and had no local dependence. During the COVID-19 epidemic, this study conducted a more comprehensive analysis of the measurement attributes of the Mandarin version of CPSS, and recommended it as a first-line stress measurement and evaluation tool for emergency medical team.

CPSS-10

10-item Chinese perceived stress scale; CTT:Classical test theory; MGRM:Multidimensional graded response model; DIF:Differential item functioning; IRT:Item response theory; SARS:Severe acute respiratory syndrome; MERS:Middle East respiratory syndrome; PSS:Perceived stress scale; PSS-10:10-item perceived stress scale; PHS:Perceived helplessness subscale; PSES:Perceived self-efficacy subscale; MIRT:Multidimensional item response theory; KMO:Kaiser Meyer Olkin; EFA:Exploratory factor analysis; CFA:Confirmatory factor analysis; ECV:Explained common variance; PUC:Percentage of uncontaminated relations; CFI:Comparative fit index; TLI:Tucker Lewis index; RMSEA:Root mean square error of approximation; SRMR:Standardized root mean square residual; Infit:Information-weighted fit statistic; Outfit:outlier-sensitive fit statistic; MNSQ:Mean square; PCA:Principal component analysis; IIF:Item information function; CI:Confidence interval.

Ethics approval and consent to participate

The study protocol was approved by the Institutional Review Boards of Weifang Medical University in March 2020 (Project approval No.: 2020YX078).

Consent for publication

Not applicable.

Availability of data and materials

All authors had full access to the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests

The author declares that he has no competing interests.

Funding

This study was partially supported by the National Natural Science Foundation of China (Project approval No.: 81872719, 81803337), the National Bureau of Statistics Foundation Project (Project approval No.: 2018LY79), the Natural Science Foundation of Shandong Province (Project approval No.: ZR2019MH034). The funders did not play any role in the study design, data collection and interpretation of data, or in writing the manuscript.

Authors’ contributions

G. Z., contributed to conceptualization, methodology, data curation, software, and original manuscript writing; Z. W., contributed to the review and editing of writing; Y. Z., contributed to data curation and methodology; J. L., contributed to data curation and the review and editing of writing; P. G., J. L., and Z. L., contributed to supervision and formal analysis; S. W. and F. S., contributed to methodology and the review and editing of writing. All authors gave final approval and agreed to be accountable for all aspects of the work.

Acknowledgments

We would like to acknowledge and thank the study participants for being involved in this project. Special thanks to Professor Zhiqiang Li of Jinzhou Medical University for providing the details of the design and implementation of the study.

Cassell G, Mekalanos J. Development of Antimicrobial Agents in the Era of New and Reemerging Infectious Diseases and Increasing Antibiotic Resistance. JAMA-J Am Med. 2001;285(5):601–5.
Guan WJ, Ni ZY, Hu Y, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. New Engl J Med. 2020;382(18):1708–20.
Zhu N, Zhang D, Wang W, et al. A Novel Coronavirus from Patients with Pneumonia in China, 2019. New Engl J Med. 2019;382(8):727–33.
Huang CL, Wang YM, Li XW, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020;395(10223):497–506.
WHO. Statement on the second meeting of the International Health Regulations(2005) Emergency Committee regarding the outbreak of novel coronavirus (2019- nCoV). 2020. https://www.who.int/news-room/detail/30-01-2020-statement-on-the-second-meeting-ofthe-international-health-regulations-(2005)-emergency-committee-regarding-the-outbreak-ofnovel-coronavirus-(2019-ncov)/. Accessed 28 June 2021.
WHO. WHO Director—General’s opening remarks at the media briefing on COVID-19. 2020. https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020 /. Accessed 28 June 2021.
WHO. WHO Coronavirus Disease (COVID-19) Dashboard. 2021. https://covid19.who.int/. Accessed 28 June 2021.
Kraemer MUG, Yang CH, Gutierrez B, et al. The effect of human mobility and control measures on the COVID-19 epidemic in China. Science. 2020;368(6490):493–7.
WHO. Director-General’s opening remarks at the media briefing on COVID-19. 2020. https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---3-march-2020/. Accessed 28 June 2021.
Shanafelt T, Ripp J, Trockel M. Understanding and Addressing Sources of Anxiety Among Health Care Professionals During the COVID-19 Pandemic. Jama. 2020;323(21):2133–4.
Greenberg N, Docherty M, Gnanapragasam S, Wessely S. Managing mental health challenges faced by healthcare workers during covid-19 pandemic. Bmj. 2020;368:m1211.
Kang L, Li Y, Hu S, et al. The mental health of medical workers in Wuhan, China dealing with the 2019 novel coronavirus. Lancet Psychiat. 2020;7(3):e14.
Xiong J, Lipsitz O, Nasri F, et al. Impact of COVID-19 pandemic on mental health in the general population: A systematic review. J Affect Disorders. 2020;277:55–64.
Salazar de Pablo G, Vaquerizo-Serrano J, Catalan A, Arango C, et al. Impact of coronavirus syndromes on physical and mental health of health care workers: Systematic review and meta-analysis. J Affect Disorders. 2020;275:48–57.
Chong MY, Wang WC, Hsieh WC, et al. Psychological impact of severe acute respiratory syndrome on health workers in a tertiary hospital. Brit J Psychiat. 2018;185(2):127–33.
Lee SM, Kang WS, Cho AR, et al. Psychological impact of the 2015 MERS outbreak on hospital workers and quarantined hemodialysis patients. Compr Psychiat. 2018;87:123–7.
Antonijevic J, Binic I, Zikic O, Manojlovic S, et al. Mental health of medical personnel during the COVID-19 pandemic. Brain Behav. 2020;10(12):e01881.
Chinese Center for Disease Control and Prevention Epidemiology Working Group for NCIP Epidemic Response. The epidemiological characteristics of an outbreak of 2019 novel coronavirus diseases (COVID-19) in China. Chin J Epidemiol. 2020;41(2):145–51.
Pappa S, Ntella V, Giannakas T, et al. Prevalence of depression, anxiety, and insomnia among healthcare workers during the COVID-19 pandemic: A systematic review and meta-analysis. Brain Behav Immun. 2020;88:901–7.
Wang LQ, Zhang M, Liu GM, et al. Psychological impact of coronavirus disease (2019) (COVID-19) epidemic on medical staff in different posts in China: A multicenter study. J Psychiatr Res. 2020;129:198–205.
Zhang WR, Wang K, Yin L, et al. Mental Health and Psychosocial Problems of Medical Health Workers during the COVID-19 Epidemic in China. Psychother Psychosom. 2020;89(4):242–50.
Lu W, Wang H, Lin Y, Li L. Psychological status of medical workforce during the COVID-19 pandemic: A cross-sectional study. Psychiat Res. 2020;288:112936.
Sheldon C, Tom K, Robin M. A Global Measure of Perceived Stress. J Health Soc Behav. 1983;24(11):385–96.
Pedrozo-Pupo JC, Pedrozo-Cortes MJ, Campo-Arias A. Perceived stress associated with COVID-19 epidemic in Colombia: an online survey. Cad Saude Publica. 2020;36(5):e00090520.
Abdel Jalil MH, Alsous MM, Hammad EA, et al. Perceived Public Stress Among Jordanians During the COVID-19 Outbreak. Disaster med public. 2020; 1–5.
Shanmugam R. Handbook of item response theory: volume one, models. J Stat Comput Sim. 2020;90(9):1922.
Barbosa-Leiker C, Kostick M, Lei M, et al. Measurement invariance of the perceived stress scale and latent mean differences across gender and time. Stress Health. 2013;29(3):253–60.
Roberti JW, Harrington LN, Storch EA. Further psychometric support for the 10-item version of the perceived stress scale. Journal of College Counseling. 2011;9(2):135–47.
Wang Z, Chen J, Boyd JE, et al. Psychometric properties of the Chinese version of the Perceived Stress Scale in policewomen. PloS one. 2011;6(12):e28610.
Barlett MS. A note on the Multiplying Factors for Various Chi Square Approximations. J R Stat Soc B. 1954;16(2):169–73.
Cerny BA, Kaiser HF. A Study of a Measure of Sampling Adequacy for Factor-Analytic Correlation Matrices. Multivar Behav Res. 1977;12(1):43–7.
Gorsuch RL. Exploratory factor analysis: its role in item analysis. J Pers Assess. 1997;68(3):532–60.
Maybery D, Jones R, Dipnall JF, et al. A mixed-methods study of psychological distress following an environmental catastrophe: the case of the Hazelwood open-cut coalmine fire in Australia. Anxiety Stress Copin. 2020;33(2):216–30.
Mitchell A, Crane P, Kim Y. Perceived stress in survivors of suicide: psychometric properties of the Perceived Stress Scale. Res Nurs Health. 2008;31(6):576–85.
Reis D, Lehr D, Heber E, Ebert DD. The German Version of the Perceived Stress Scale (PSS-10): Evaluation of Dimensionality, Validity, and Measurement Invariance With Exploratory and Confirmatory Bifactor Modeling. Assessment. 2019;26(7):1246–59.
Rodriguez A, Reise SP, Haviland MG. Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures. J Pers Assess. 2016;98(3):223–37.
Hu LT, Bentler PM. Fit indices in covariance structure modelling: sensitivity to underpararmeterized model misspecification. Psychol Methods. 1998;3:424–53.
Taylor JM. Psychometric analysis of the Ten-Item Perceived Stress Scale. Psychological Assessment. 2015; 27(1).
Visser MS, Dieleman M, Klijn S, et al. Validation, test-retest reliability and norm scores for the Dutch Catquest-9SF. Acta Ophthalmol. 2017;95(3):312–9.
Wu SM, Amtmann D. Psychometric Evaluation of the Perceived Stress Scale in Multiple Sclerosis. ISRN Rehabilitation. 2013; 2013:1–9.
Chaaya M, Osman H, Naassan G, Mahfoud Z. Validation of the Arabic version of the Cohen Perceived Stress Scale (PSS-10) among pregnant and postpartum women. BMC Psychiatry. 2010;10:111.
Wongpakaran N, Wongpakaran T. The Thai version of the PSS-10: An Investigation of its psychometric properties. Biopsychosoc Med. 2010;4:6.
Siu-Man N. Validation of the 10-item Chinese perceived stress scale in elderly service workers: one-factor versus two-factor structure. BMC Psychol. 2013;1(1):1–9.
Park SY, Colvin KF. Psychometric properties of a Korean version of the Perceived Stress Scale (PSS) in a military sample. BMC Psychol. 2019;7(1):58.
Andreou E, Alexopoulos EC, Lionis C, et al. Perceived Stress Scale: reliability and validity study in Greece. Int J Env Res Pub He. 2011;8(8):3287–98.
Wiriyakijja P, Porter S, Fedele S, et al. Validation of the HADS and PSS-10 and a cross-sectional study of psychological status in patients with recurrent aphthous stomatitis. J Oral Pathol Med. 2020;49(3):260–70.
Perera MJ, Brintz CE, Birnbaum-Weitzman O, et al. Factor structure of the Perceived Stress Scale-10 (PSS) across English and Spanish language responders in the HCHS/SOL Sociocultural Ancillary Study. Psychol Assess. 2017;29(3):320–8.
Crutzen R, Peters GY. Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all. Health Psychol Rev. 2017;11(3):242–7.
Robinson M, Johnson AM, Walton DM, MacDermid JC. A comparison of the polytomous Rasch analysis output of RUMM2030 and R (ltm/eRm/TAM/lordif). BMC Med Res Methodol. 2019;19(1):36.
Tu WJ, Waddimba AC, Mohr DC, et al. Physicians’ perceptions of autonomy support during transition to value-based reimbursement: A multi-center psychometric evaluation of six-item and three-item measures. PloS one. 2020;15(4):e0230907.
Teresi JA, Ocepek-Welikson K, et al. Evaluation of the measurement properties of the Perceived Stress Scale (PSS) in Hispanic caregivers to patients with Alzheimer's disease and related disorders. Int Psychogeriatr. 2020;32(9):1073–84.
Cohen S, Williamson GM. Peceived stress in a probability sample if the United States. In: Spacapan S, Oskamp S, editors. The social psychology of health. Newbury Park: Sage; 1988.
Hagquist C, Bruce M, Gustavsson JP. Using the Rasch model in nursing research: an introduction and illustrative example. Int J Nurs Stud. 2009;46(3):380–93.
Fassaert T, De Wit MA, Tuinebreijer WC, et al. Psychometric properties of an interviewer-administered version of the Kessler Psychological Distress scale (K10) among Dutch, Moroccan and Turkish respondents. Int J Methods Psychiatr Res. 2009;18(3):159–68.

Additionalfile1.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Development and Validation of the Perceived Stress Scale in Emergency Medical Teams During the Epidemic of COVID-19

Status:

Version 1

Abstract

Figures

Introduction

Methods

Study design and participants

Measures And Instrument

Factor Variables Of Differential Item Functioning

Statistical Analysis

Results

Exploratory factor analysis (EFA)

Confirmatory Factor Analysis

Unidimensionality

Reliability

Multidimensional Graded Response Model (Mgrm) Analysis

Expected Score And Item Information Function Of Cpss-10

Differential Item Functioning (Dif)

Revision Of Cpss-10 Scale

Discussion

Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1