Development and Validation of a Multi-lingual Online Questionnaire for Surveying the COVID-19 Prevention and Control Measures used in Global Workplaces

doi:10.21203/rs.3.rs-957385/v1

Background Despite widespread COVID-19 vaccination programs, there is an ongoing need for targeted disease prevention and control efforts in high-risk occupational settings. This study aimed to develop, pilot, and validate an instrument for surveying occupational COVID-19 infection prevention and control (IPC) measures available to workers in diverse geographic and occupational settings.

Methods A 44-item online survey was developed, translated, and validated for face, content, and cross-cultural validity according to literature review, expert consultation, and pre-testing. The survey was piloted with 890 workers from diverse industries in Canada, Ireland, Argentina, Poland, Nigeria, China, the US, and the UK. Odds ratios generated from univariable, and multivariable logistic regression assessed differences in ‘feeling protected at work’ according to gender, age, occupation, country of residence, professional role, and vaccination status. Exploratory factor analysis (EFA) was conducted, and internal consistency reliability verified with Cronbach’s alpha. Hypothesis testing using two-sample t-tests verified construct validity (i.e., discriminant validity, known-groups technique), and criterion validity.

Results After adjustment for occupational sector, characteristics associated with feeling protected at work included being male (AOR = 1.88; 95% CI = 1.18,2.99), being over 55 (AOR = 2.17; 95% CI = 1.25,3.77) and working in a managerial position (AOR = 3.1; 95% CI = 1.99,4.83). EFA revealed nine key IPC domains relating to: environmental adjustments, testing and surveillance, education, costs incurred, restricted movements, physical distancing, masking, isolation strategies, and areas for improvement. Each domain showed sufficient internal consistency reliability (Cronbach’s alpha ≥ 0.60). Hypothesis testing revealed differences in survey responses by country and occupational sector, confirming construct validity (p < 0.001), criterion validity (p = 0.04), and discriminant validity (p < 0.001).

Conclusions The online survey, developed to identify the COVID-19 protective measures used in diverse, international workplace settings, showed strong face validity, content validity, cross-cultural validity, internal consistency, criterion validity, and construct validity. It can be used by decision makers in the distribution of IPC resources, and to guide occupational safety and health (OSH) recommendations for preventing COVID-19 and future infectious disease outbreaks.

Health Economics & Outcomes Research

Health Policy

Occupational safety and health

COVID-19

psychometric validation

online survey

infection prevention and control

Relatively few infectious individuals (~10%) are responsible for most (~80%) of local COVID-19 transmission (1). Environmental factors likely contribute to COVID-19 clusters, or ‘superspreading’ events. Crowded indoor settings, poor ventilation, and long contact exposures create high-risk working environments for COVID-19 (2). Workplaces such as hospitals, care homes, building sites, food-processing plants, and retail outlets possess such environmental features that facilitate explosive COVID-19 clusters (3). Comprehensive occupational COVID-19 infection prevention and control (IPC) measures can prevent these events, ensuring the protection of workers, their families, and surrounding communities. Effective measures include swift and thorough contact tracing and case isolation, effective personal protective equipment (PPE), testing, worker bubbles, and improved ventilation and air quality (4,5). Masking alone is not enough to prevent COVID-19 clusters in high-risk workplace environments (5).

Many countries have been active in creating COVID-19 occupational safety and health (OSH) policies and guidance involving these measures (6). Nevertheless, data indicate that workers remain at disproportionate risk to COVID-19 in settings where vaccination rates lag and cases surge (7–9). The United States’ Occupational Safety and Health Administration (OSHA) issued its first mandatory workplace safety rules aimed at protecting healthcare workers from COVID-19 in June 2021 (10). Yet these rules do not protect workers in other essential industries who face rising case numbers, stalled vaccinations (11), loosened mask requirements (12), and push back against occupational COVID-19 safety regulations from the national business community (13). In Europe, as businesses reopen and travel restrictions ease (14), countries continue to experience workplace outbreaks (7,8). Case numbers amongst oil sands workers in Canada grew so high in May 2021 that legislators had to declare a state of local emergency (15); and in India, an unprecedented surge in cases March-April 2021 strained IT firms who struggled to protect the health of employees while maintaining business continuity (16). These events highlight the ongoing need for disease prevention and control guidance and resources directed towards workers in high-risk occupational settings.

The International Labour Organization (ILO) has compiled international occupational policy responses to the COVID-19 crisis (17). To our knowledge, no such effort has been undertaken to survey the extent to which these policies are being implemented from the perspective of workers. As research reveals gaps in policymakers’ and workers’ perceptions of acceptable COVID-19 safety at work, it is important to constructively engage with workers in order to fast track acceptable and feasible OSH solutions (18). Understanding protective measures in place can, by helping to untangle potential causes of outbreaks, aid governments trying to contain or prevent COVID-19 surges around the world. Identifying gaps in workplace safety response can help to direct the distribution of disease prevention and control resources and raise awareness of effective COVID-19 safety measures.

This study aimed to develop, pilot, and validate an instrument for identifying occupational COVID-19 IPC measures available to workers from diverse geographic and occupational settings. By allowing researchers to measure workers’ perceptions of the level of COVID-19 protections in place, the survey instrument will have important implications for worker protection from COVID-19 and future infectious disease outbreaks.

This cross-sectional, pilot study was conducted in three phases: 1) survey development, 2) pilot testing, and 3) psychometric property evaluation. Psychometric properties assessed as part of the study are defined in Figure 1. The methodological steps taken to validate them and finalize the survey are outlined in Figure 2.

Phase 1. Survey Development

Creating the survey

From February to April 2019, the researchers conducted an extensive review of relevant research literature and meta-analysis to investigate the effectiveness of measures to prevent and control COVID-19 outbreaks in global workplace settings. Results of this review identified that combinations of (1) basic preventive measures (i.e., masking, hand hygiene, and social distancing), (2) surveillance measures, (3) outbreak investigations and response, (4) environmental adjustments, and (5) education initiatives can effectively prevent workplace outbreaks (5). These categories became the framework for the development of a survey allowing researchers to measure the level of occupational COVID-19 protection available to workers from diverse geographic and occupational settings.

A preliminary questionnaire in English was designed on Qualtrics^XM (Provo, UT) online survey platform with questions pertaining to:

Basic Demographics (age, gender, country of residence, education);
Employment Demographics (workplace category, industry, size, current role);
Basic preventive measures (hand hygiene, barriers, social distancing, masking, etc.);
Environmental adjustments (temperature, ventilation, air quality monitoring);
Surveillance (type of testing, access to testing, self-isolation requirements, syndromic surveillance);
Contact tracing;
PPE;
Education and training; and
Vaccination.

Questions were designed for actively working employees, managers, and occupational safety and health professionals from any country. Skip patterns were enabled based on participant characteristics. For example, managers and employees were given access to separate question tracks, and only participants that selected “Healthcare and Social Assistance” as their occupational sector could access questions on medical PPE. As the survey was intended for internationally and occupationally diverse participants, survey questions were worded as generally and simply as possible.

Establishing content validity

The preliminary survey was independently reviewed by five multinational experts in Public Health and Occupational Safety and Health to ensure face validity (that the questions adequately met the study’s aims) and content validity (to ensure that questions were all relevant to the study’s aims). Once independent review was complete, experts convened for two, one-hour meetings with research team members to discuss findings. Survey items that did not reach consensus during the first meeting were removed, modified, or reworded by the researchers and presented again for expert review. During the second meeting, experts reached consensus on all survey items as well as on general format, language, and response options.

Prior to dissemination, the Qualtrics survey was pre-tested with 10 actively working individuals from Ireland and the United States. Researchers met one-one-one with construction (n=1), retail (n=2), education (n=3), manufacturing (n=1), and healthcare (n=3) employees to ensure adequate comprehension of the information sheet, survey items, and response options. Pre-testing data was reviewed, and the survey adapted based on participant feedback. Researchers added questions pertaining to a pandemic transition phase (i.e., measures that workers would like to see upheld/removed as vaccinations increase), and whether or not workers feel adequately protected from COVID-19 at work. Pre-testing allowed researchers to identify and correct errors in skip patterns.

Translating the survey

The survey instrument, once finalized in English, was translated into Spanish, French, Polish, Hindi, and Chinese. Members of the research team with native-level fluency in these languages conducted forward translation. Subsequent back-translation using Google Translate^TM allowed researchers to identify and fix any discrepancies. All translated versions of the survey were pre-tested with at least one actively working native speaker to ensure ease of comprehension of translated items.

Phase 2. Pilot Testing

Participant recruitment

Participants were recruited between 01 July and 01 August 2021 through non-probability convenience sampling techniques. Included participants were working individuals aged 18 or older with literacy in English, French, Spanish, Hindi, Polish, or Chinese. Participants who were not actively working or who were currently working full-time from home were not excluded from participating. Instead, they were directed to a shortened version of the questionnaire with questions pertaining to (1) basic demographics, (2) employment demographics, and (3) vaccination status. Because working from home is a safety measure in itself, the inclusion of said participants was considered relevant to the survey’s objectives.

Workers in countries where the research team had professional contacts and/or where appropriate survey translations were available were targeted in this pilot study. To ensure a geographically and occupationally diverse sample, researchers advertised the survey link through multiple recruitment channels in a manner similar to McRobert et al. (19):

Formal recruitment channels: OSH organizations and national trade unions were contacted by the researchers and, if willing to collaborate, sent the survey link to all constituents via email;
Social media groups: Researchers joined relevant occupational LinkedIn and Facebook groups and posted the survey link into those groups; and
Personal social media: Researchers’ posted the link to their own social media profiles (Facebook, LinkedIn, Twitter, WhatsApp).

The multi-modal online participant recruitment strategy was approved by the University College Dublin Human Research Ethics Committee (LS-E-21-138-Perrotta) and followed recommended procedures for online surveys. By keeping careful records of where, when, and by whom participants were contacted, and the number of participants contacted (20), partnering with other organizations and encouraging peer-led snowball sampling (21), and using an open access, ethically approved, and user optimized instrument (22), the researchers limited potential for sample bias (20,23). Professional collaborators who agreed to advertise the survey included the European Agency for Safety and Health at Work (EU-OSHA), the University College Dublin Centre for Safety and Health at Work, and the British Columbia General Employees’ Union. The survey link was posted in 22 international, OSH-themed LinkedIn and Facebook groups accounting for 270,000 total group members.

Survey results

To assess associations between participant characteristics and workplace protection, odds ratios (OR) and adjusted odds ratios (AOR) generated from logistic regression models were used to estimate the likelihood of feeling protected at work vs. feeling unprotected or unsure according to age, gender, country of residence, education level, occupation, and vaccination status. Stepwise model selection by Akaike information criterion (AIC) was performed to determine the best-fit multivariable model using R version 4.0.2 step() function (R Foundation for Statistical Computing, Vienna, Austria). Survey data used for logistic regression is provided in Additional File 1.

Phase 3. Psychometric Property Evaluation

Quantitative survey validation followed recommended guidelines for survey reliability and validity testing (24). All data were analysed in R. Note that content and cross-cultural validity were tested prior to data collection during the survey development phase. Survey reliability, criterion and construct validity were evaluated following data collection.

Reliability:

Prior to data collection, the researchers hypothesized that preventive constructs fell under seven IPC measures’ domains: basic preventive measures, environmental adjustments, surveillance, contact tracing, PPE, education and training, and vaccination. To test this, exploratory factor analysis (EFA) was conducted using R package ‘psych’ 2.1.6 to identify the underlying factor structure explaining the relationship between 44 measured variables (25). Questions relating to medical PPE and contact tracing were excluded from EFA due to large quantities of missing data. To determine the suitability of data for EFA, Pearson correlation matrices were verified for a statistically significant Bartlett’s test and a Kaiser-Meyer-Olkin (KMO) statistic above 0.60 (25). Scree plot inspections and parallel analysis based on minimum rank factor analysis (PA-MRFA) were conducted to determine the advised number of factor dimensions. The PA-MRFA method, which expresses model fit by reporting the overall percentage of common variance explained (25), has been recommended for assessing the number of common factors underlying polytomous variables (26). The Oblimin oblique rotation (non-orthogonal) method was used to aid factor interpretation, with rotated loadings above 0.32 considered acceptable (27). Data and codes used for EFA, and a description of coded response options are available in Additional Files 2-3.

Cronbach’s alpha coefficients and values of alphas if removed were calculated to measure internal consistency within the IPC domains identified through EFA. Values ≥ 0.6 indicated acceptable internal consistency – that is, that all questions within the domain measured the appropriate protective construct – a threshold that has been recommended in the early stages of research (28,29).

Criterion Validity:

Criterion validity is the relation between the score of an instrument and, conventionally, another instrument that is widely accepted as a ‘gold standard’ (24). Because this was the first survey intended to measure the level of COVID-19 protection in a participants’ workplace, no accepted instrument existed for comparison. Instead, researchers asked participants to identify whether or not they felt protected in the workplace at the end of the survey, determining whether investigated protective measures related to participants’ own standards for adequate safety. This method followed a similar procedure to Watkins et al. who looked for an alternative to a long instrument to assess depression and tested a single question – Do you frequently feel sad or depressed (30)? This study sought to verify a long instrument to assess COVID-19 workplace safety by comparing it to a single question – Do you feel protected from COVID-19 at work?

To test for criterion validity, an overall COVID-19 IPC measures’ protective score was calculated out of 40 possible points and compared to the survey’s ‘gold standard’ question. Respondents received one point for each affirmative response to questions on basic preventive measures, environmental adjustments, testing and surveillance, education and training, PPE, contact tracing, vaccination status, and access to paid sick leave. Detailed score calculations are provided in Additional File 4. Protective scores were compared to whether or not a worker felt protected from COVID-19 at work using two-sample t-tests; the significance level was set at p < 0.05.

Construct Validity:

To understand the extent to which a set of variables represent the construct intended to be measured (i.e., construct validity), discriminant validity tests the hypothesis that a target-measurement is not improperly related to variables from which it should differ (31). As the survey aimed to measure the level of occupational COVID-19 protection available to workers, a higher number of IPC measures in place should not be associated with feeling unprotected at work and vice versa. To test this, respondents were divided into quartiles according to total protective measures scores. Odds ratios (OR) generated from univariable logistic regression were used to estimate the likelihood of feeling protected at work according to protective score quartile. Known-groups technique was applied to assess whether the survey instrument successfully captured expected differences between groups of survey respondents. Hypotheses were formulated prior to data collection and tested using two-sample t-tests (significance level: p < 0.05).

Participant characteristics and survey results

A total of 1,473 eligible participants viewed and clicked on the survey link. Participants were given access to the study information sheet from the survey home page. Informed consent to participate was requested before the survey could begin. A total of 890 surveys were completed giving a response rate of 60.4%. The survey took 6.5 minutes to complete on average (min: 1.9 – max: 16.5). 70% (627/890) of respondents were actively in the workplace (i.e., not working from home or out of work) and included in psychometric property evaluation. A description of active workers’ socio-demographic and occupational characteristics is displayed in Table 1 along with univariable and multivariable logistic regression results for feeling protected at work vs. feeling unprotected or unsure.

Table 1

Participant characteristics and logistic regression results for feeling protected from COVID-19 at work vs. feeling unprotected or unsure (N=627)
		N	%	Do you feel protected from COVID-19 at work? Yes vs. No or unsure Crude OR (95% CI)	Adjusted OR (95% CI)^c
Gender	Female^a	436	71
	Male	181	29	1.99 (1.35,2.94)***	1.88 (1.18,2.99)**
	Total	617	100
Age	18 to 34^a	151	24
	35 to 44	165	26	1.05 (0.66,1.67)	1.29 (0.76,2.21)
	45 to 54	177	28	1.5 (0.94,2.40)	1.64 (0.98,2.77)
	55 and over	134	21	1.84 (1.10,3.05)*	2.17 (1.25,3.77)**
	Total	627	100
Country of Residence^b	Ireland ^a	210	36
	Argentina	24	4	2.02 (0.71,5.76)
	Canada	243	42	0.80 (0.54,1.18)
	China	26	4	2.15 (0.76,6.07)
	Nigeria	18	3	1.16 (0.41,3.27)
	Poland	16	3	1.05 (0.37,3.02)
	UK	49	8	0.40 (0.20,0.78)**
	Total	586	100
Education Level	No College Education ^a	146	23
	College Education or higher	481	77	0.71 (0.47,1.06)
	Total	627	100
Occupational Sector	Healthcare or social assistance^a	255	50
	Construction	15	3	1.18 (0.39,3.55)	0.98 (0.23,4.23)
	Educational services	106	21	0.48 (0.30,0.79)**	0.74 (0.26,2.13)
	Administration	19	4	0.65 (0.26,1.67)	2.03 (0.75,5.53)
	Manufacturing and food processing	15	3	7.64 (0.98,59.43)	6.58 (0.67,64.61)
	Professional, scientific, or technical services	31	6	1.85 (0.76,4.50)	2 (0.55,7.25)
	Public administration and defence	25	5	0.82 (0.35,1.93)	1.37 (0.37,5.04)
	Transportation or warehousing	17	3	0.46 (0.16,1.27)	0.44 (0.09,2.1)
	Retail trade	27	5	0.23 (0.09,0.57)**	0.38 (0.1,1.46)
	Total	509	100
Role	Employee^a	432	69
	Management	195	31	2.99 (2.01,4.43)***	3.1 (1.99,4.83)***
	Total	627	100
Vaccination Status	Vaccinated (Fully or Partially) ^a	460	73
	Unwilling to be vaccinated	62	10	0.82 (0.48,1.43)
	No access to COVID vaccine	105	17	0.57 (0.34,0.95)*
	Total	627	100
p < 0.05, p < 0.01, **p < 0.001
^a Reference category
^b Because this study aimed to determine the survey’s ability to capture country-level differences in response, we chose not to combine groups despite low percentages of responses from some countries (~3%)
^c Stepwise descending variable selection using AIC

Results from univariable logistic regression showed that men were more likely to feel protected at work than women (OR = 1.99; 95% CI = 1.35,2.94), and that older workers (> 55 yrs.) were more likely to feel protected at work than young workers (< 35 yrs.) (OR = 1.84. 95% CI = 1.10,3.05). Though 78% (453/586) of survey responses yielded from Ireland and Canada, responses from Argentina, China, Nigeria, Poland, and the UK were also collected. Respondents from the UK were significantly less likely to feel protected at work than Irish respondents (OR=0.40; 95% CI =0.20,0.78). Half of survey respondents were healthcare workers (255/509); the remaining half yielded from diverse occupational sectors. Educators felt significantly less protected at work than healthcare workers (OR = 0.48; 95% CI = 0.30,0.79), as did retail workers (OR = 0.23; 95% CI = 0.09,0.57). Managers, accounting for 31% of respondents (195/627), were significantly more likely to feel protected than employees (OR = 2.99; 95% CI = 2.01,4.43). Compared to the 73% (460/627) of vaccinated respondents, having no access to a COVID-19 vaccine left workers feeling unprotected (OR=0.57; 95% CI =0.34,0.95). Choosing not to be vaccinated was not significantly associated with feeling unprotected at work (OR = 0.82; 95% CI = 0.48,1.43). In the adjusted model, men (AOR = 1.88; 95% CI = 1.18,2.99), over 55’s (AOR = 2.17; 95% CI = 1.25,3.77), and managers (AOR = 3.1; 95% CI = 1.99,4.83) remained significantly more likely to feel protected at work after adjustment for gender, age, professional role, and occupational sector.

Table 1. Participant characteristics and logistic regression results for feeling protected from COVID-19 at work vs. feeling unprotected or unsure (N=627)

Protective measure scores out of 40 were calculated for all actively working respondents (n=627) based on the number of safety measures identified in their workplace. Participants scored 21 points on average (min: 9 – max: 36). Average protective measures scores are displayed by country and occupational sector in Figure 3.

Survey Validation Results

Survey data was determined suitable for EFA with evidence of substantial correlations between items, Bartlett’s statistic = 7584.4 (df = 946), p <0.001, and KMO = 0.77. Scree plots and PA-MRFA suggested a 9-factor model accounting for 67.25% of common variance explained. Items for which loadings were non-significant (< 0.32) were dropped from respective factors. Internal consistency results for the nine identified IPC measures survey domains are shown in Table 2. All domains satisfied internal reliability criteria (Cronbach α ≥ 0.60). Masking and social distancing domains had the highest average inter-item correlations (Cronbach α = 0.90, 0.97); domains relating to costs incurred and limiting contacts had the lowest (Cronbach α = 0.60). Questions on vaccination status, access to a contact tracing program, handwashing, and use of signage in the workplace did not fit into identified IPC domains but were kept as separate constructs to ensure content validity. Questions on the need to improve vaccination uptake amongst employees, the use of plastic barriers, and masking of non-employees were removed from the survey.

Table 2

Internal consistency of COVID-19 IPC Measures Survey Domains as indicated by Cronbach’s alpha (n=627 actively working respondents)
COVID-19 IPC Measures Domain^a	Items^b	α if item removed^c
Areas for Improvement α = 0.73 ^c	Contact tracing	0.69
	Education and training	0.69
	Access to protective materials	0.69
	Testing	0.70
	Implementation of basic preventive measures by management	0.70
	Adherence to basic preventive measures by employees	0.70
	Funding for safety measures	0.72
	Environmental conditions	0.72
	Access to vaccines	0.73
Environmental Adjustments α = 0.66 ^c	Temperature adjustments	0.56
	Ventilation adjustments	0.65
	Environmental monitoring	0.61
	Air quality monitoring	0.56
Testing and Surveillance α = 0.62 ^c	Testing of symptomatic employees	0.48
	Testing of close contacts	0.51
	Employee temperature checks	0.62
	Symptom reporting to management	0.57
Employee Education and Training α = 0.68 ^c	Training on proper PPE use	0.66
	Training on how to safely interact with colleagues in work	0.49
	Training on how to safely socialize outside of work	0.61
Costs Incurred α = 0.60 ^c	Free COVID-19 testing	0.54
	Paid time off for testing	0.54
	Paid sick leave	0.54
Number of Contacts α = 0.60 ^c	Worker movement restricted between facilities	0.45
	Worker bubbles	0.51
	Only essential personnel enter facility	0.57
	High-risk employees (> 60 yrs, underlying health conditions) stay at home	0.58
Physical Distancing α = 0.97 ^c	Physical distancing rules in place	0.95
Physical Distancing α = 0.97 ^c	Physical distancing is maintained	0.92
Isolation Strategies α = 0.73 ^c	Confirmed COVID-19 cases self-isolate	0.56
	Symptomatic employees self-isolate	0.54
	Close contacts self-isolate	0.78
Masking α = 0.90 ^c	All employees are instructed to wear masks	0.83
Masking α = 0.90 ^c	Masks are worn correctly in the workplace	0.81
^aDomains identified through exploratory factor analysis
^b Items coded as: Yes (2), Unsure (1), No (0)
^c Standardized alpha’s reported

Hypotheses defined prior to data collection and results from criterion and construct testing are displayed in Table 3. To assess criterion validity, protective measures’ scores were compared to the survey’s ‘gold-standard’ question: Do you feel protected from COVID-19 in the workplace? A significant association was identified using two-sample t-tests (p < 0.001), confirming a relation between a participants’ survey results and feeling protected at work. More precisely, to determine discriminate validity, participant protective scores were divided into quartiles ranging from highest protection (23-36 total measures in place] to lowest (≤ 17 total measures). Results from univariable logistic regression showed that participants in the highest protective quartile were significantly more likely to feel protected from COVID-19 at work than those in the lowest quartile (OR = 2.67; 95% CI = 1.17,2.95); and that participants in the lowest quartile were significantly less likely to feel protected than participants in the highest quartile (OR = 0.38; 95% CI = 0.23,0.62).

Table 3

Criterion and construct validity of COVID-19 IPC Measures Survey (n=627 actively working respondents)
Validity Measure	Hypothesis tested	n	Test used	Result	Conclusion
Concurrent validity	There is a positive association between overall protective measures score and the ‘gold-standard’ question: Do you feel protected from COVID-19 at work?	583^a	Two-sample t-test^b	p < 0.001 ***	The survey instrument successfully measures the degree of COVID-19 safety a participant feels in the workplace.
Known-groups technique	The number of protective measures in place varies significantly by country	453^c	Two-sample t-test^bc	p = 0.04 *	The survey instrument successfully captures national differences in workplace protective response
Known-groups technique	The health care and social assistance sectors have more protective measures in place than other occupational sectors	627	Two-sample t-test^b	p <0.001***	The survey instrument successfully captures occupational differences in workplace protective response
Discriminant validity	A participant in the highest protective scores quartile (i.e., whose workplace has more protective measures in place than 75% of survey respondents) is significantly more likely to feel protected at work than a participant in the lowest protective scores quartile	583 ^a	Univariable logistic regression^f	OR (95% CI) = 2.67 (1.17,2.95)***	Respondents with high IPC coverage are more likely to feel protected at work than those with low IPC coverage. The survey instrument is not improperly measuring the level of protection available to workers.
Discriminant validity	A participant in the lowest protective scores quartile (i.e., whose workplace has fewer protective measures in place than 75% of survey respondents) is significantly less likely to feel protected at work than a participant in the highest protective scores quartile	583 ^a	Univariable logistic regression^d	OR (95% CI) = 0.38 (0.23,0.62)***	Respondents with low IPC coverage are less likely to feel protected at work than those with high IPC coverage. The survey instrument is not improperly measuring the level of protection available to workers.
p < 0.05, p < 0.01, **p < 0.001
^a Data on ‘Do you feel protected from COVID-19 at work?’ missing for n=44 participants
^b Heterogeneity of variances verified- Bartlett test; normality verified -Shapiro-Wilks statistic
^c Two largest participant groups compared: Canada (n=243) and Ireland (n=210)
^d Dependent variable: Feeling protected at work vs. feeling unprotected or unsure

Construct validity was assessed using known-groups technique, whereby expected differences between groups of respondents were tested and verified. Significant differences in average protective score were detected between Irish and Canadian respondents, the two largest participant groups, according to two-sample t-tests (p=0.04). Health care and social assistance sectors had more protective measures in place than other occupational sectors (p < 0.001), as hypothesized. Results displayed in Table 2 further demonstrate the survey’s ability to distinguish between groups. Feeling protected varied significantly by country (i.e., UK respondents were less likely to feel protected at work than Irish respondents), and by occupational sector (i.e., retail workers and educators were less likely to feel protected than healthcare workers).

The final survey instrument based on pre-testing and psychometric property evaluation is available in Additional File 5.

This survey was developed to measure the level of occupational COVID-19 protection available to workers from diverse geographic and occupational settings. Following expert consultation and pre-testing, the survey showed strong face and content validity. Psychometric property evaluation revealed satisfactory levels of internal consistency reliability, and criterion and construct validity. The survey provides a framework for evaluating the level of protection a worker perceives they have from COVID-19 in the workplace. It holds the potential to be of particular use to decision-makers in the distribution of infectious disease prevention and control resources, and to guide recommendations for keeping the global workforce safe and healthy at the onset of emerging infectious disease outbreaks.

Exploratory factor analysis revealed nine key IPC domains that were confirmed by a high percentage of common variance explained. Cronbach’s alpha coefficients showed that items in each domain measured the appropriate construct. By providing a more precise model than originally hypothesized by the researchers, EFA provided insight into how occupational IPC guidelines and programs can be structured. For example, limiting worker movements and contacts emerged as a separate construct from physical distancing between workers. Testing and isolation measures emerged as separate from the cost of testing and isolation incurred. Survey items on universal testing, contact tracing, vaccinations, and signage did not fit into the 9-domain model. However – because research shows the ability of universal testing and timely contact tracing to prevent COVID-19 outbreaks (5); that the rollout of vaccines plays a crucial part in protecting workers (32); and that COVID-19 training should be reinforced by the use of signage placed in strategic locations (33) – each of these items were considered important for content validity and left in the survey.

By design, this study focused on survey development and validation. Nevertheless, survey results and hypothesis testing applied in the context of psychometric property evaluation revealed important considerations for occupational health. In terms of country-level differences, Canadian workers identified a higher number of implemented COVID-19 IPC precautions than Irish respondents despite representing a lower percentage of health care workers (42% of Canadian respondents worked in health care or social assistance vs. 60% of Irish respondents). This may be explained, in part, by an older sample of Canadian respondents. Research shows that older and more experienced workers report greater comfort with IPC skills (34), corroborating our finding that older workers were twice as likely to feel protected as younger workers. However, with both countries reporting similar vaccination rates and the implementation of return-to-work COVID-19 safety policies (6, 11), it will be important to examine survey data for specific variations in workplace IPC response. Though the total number of protective measures did not vary significantly between UK and Irish respondents, UK respondents reported feeling significantly less protected at work. One explanation relates to measures’ adherence. For example, over one third of UK respondents reported that colleagues ‘rarely’ or ‘never’ wore masks correctly, compared to just 10% of Irish respondents. This demonstrates how individuals may perceive and prioritize safety differently, particularly when COVID-19 safety precautions have the potential to create physical or social discomfort (6, 35), add to the workload (34), or result in a loss of income (32% of survey respondents reported not receiving paid time off to be tested for COVID-19). Consequently, voluntary adherence to instituted IPC measures will require clear and systematic communication on the necessity for and strategy in place behind them (36). Though relatively few workers responded from developing countries, we chose to report findings from Nigeria and Poland to demonstrate the importance of targeting workers in less-studied, low- and middle-income countries in future survey distribution. Nigerian participants reported some of the lowest numbers of protective measures despite a surge in cases at the time of survey distribution, indicating that factors such as financial resources and public health infrastructure may be contributing to differences in occupational COVID-19 response.

Feeling protected at work varied according to gender, professional role, and occupational sector. In line with reports that women are more likely to feel concerned over their own health and the health of others’ (37), survey results indicated that men felt significantly more protected at work than women. The COVID-19 pandemic has exacerbated gender inequalities. Despite being more likely to work on the frontline and in essential sectors, women are severely underrepresented in COVID-19 decision-making bodies (38). We found that twice as many male respondents were in managerial roles as female respondents, and that managers felt more protected than employees after accounting for sector effect. Educators and retail workers felt significantly less protected than healthcare workers, with health care workers reporting the highest average protective score and retail workers the lowest. While IPC response has been widely studied in the healthcare setting (5), few studies have examined COVID-19 safety amongst retail, transport, and other essential workers. Our finding that 55% (197/359) of non-health care respondents felt unprotected at work, coupled with reports of COVID-19 outbreaks in international bars, building sites, food-processing plants, hotels, shops, and transportation settings (3), underline the value in studying the COVID-19 safety resources available to geographically and occupationally diverse groups of workers. By successfully capturing gender and occupational inequalities, the survey is an effective tool for tailoring disease prevention and control response to those most in need of protection.

Similar to findings from Ireland and Canada (39, 40), 10% of survey respondents reported an unwillingness to be vaccinated for COVID-19. This group did not feel significantly less protected at work than vaccinated participants. Misinformation is likely to be contributing to a sense of false security. Research shows that vaccine-hesitant individuals consume significantly less information about COVID-19 from newspapers, television, radio, and government agencies, reporting higher levels of trust in social media (41). Recommendations for addressing vaccine hesitancy include collaborations between health agencies and multiple societal stakeholders to avoid the feeling that public health measures are for the benefit of government authorities (42); and public health messaging that is targeted, clear, direct, repeated, positively orientated, and emphasises the personal benefits of vaccination against COVID-19 (41). Similar recommendations can be applied in the workplace to ensure adequate awareness of and adherence to occupational safety precautions that are important in addition to vaccination campaigns. Of 456 vaccinated survey respondents, 38% still did not feel adequately protected from COVID-19 at work.

Pre and pilot testing resulted in additional lessons learned regarding survey distribution. The multi-modal recruitment strategy used resulted in greater uptake than reported in other, similar studies (McRobert et al. received 387 complete responses from a multidisciplinary international sample of clinicians using social media posts and email distribution over a three-month period (19)). Nevertheless, we found that posting in relevant occupational LinkedIn groups garnered little to no response. Posting in occupational Facebook groups was slightly more successful, however researchers were required to post from personal profiles creating a level of individual exposure not identified by other studies that used Facebook groups to recruit for health research (43, 44). The most successful recruitment method involved partnering with relevant organizations who could distribute the survey link via their email network (e.g., EU-OSHA, UCD Centre for Safety and Health at Work, BC General Employees’ Union). However, ensuring partner participation required extensive effort. Of 132 emails sent by the researchers to international trade unions and occupational safety and health organizations, seven eventually agreed to distribute the survey (5%). Reasons for declined participation included precarious political and/or pandemic-related circumstances, overworked staff, and survey fatigue. As a next step, and to ensure as robust and unbiased a study sample as possible, the research team plans to roll out the survey in select target countries using Qualtrics^XM Research Services; to post the survey link to personal social media profiles and encourage snowballing with the aid of a promotional video (Additional File 7); and to continue working with external partners to solicit participation via email recruitment.

Limitations

This study has several limitations. Because all survey data was collected anonymously via Qualtrics^XM survey platform, test-retest reliability could not be assessed. Despite efforts to gather pilot data from as representative sample as possible, survey uptake was greater in Ireland and Canada than in other targeted countries. Workers in high-risk occupational settings like manufacturing and food-processing, retail, transport, and construction also proved difficult to recruit. Though the survey was translated into multiple languages and verified for cross-cultural validity, most survey responses were based on the English version of the survey. Psychometric property evaluation is therefore less relevant to translated versions. Despite these limitations, pilot data allowed for sufficient psychometric property evaluation in English. The researchers have developed strategies for a more equitable roll-out of the validated survey instrument moving forward.

A multi-lingual online survey to measure the level of occupational COVID-19 protection available to workers in diverse geographic and occupational settings was developed, piloted, and validated. The survey showed strong face and content validity, as well as internal consistency, criterion validity, and construct validity. Translated versions of the survey into Polish, Spanish, French, Hindi, and Chinese showed cross-cultural validity. The survey instrument is suitable for measuring occupational COVID-19 IPC response in varied global and workplace settings, providing valuable insight for COVID-19 occupational safety and health guidelines and future infectious disease preparedness. As a follow-up to this research, survey data will be used to map and compare occupational COVID-19 IPC measures used in international workplace settings.

AIC	Akaike information criterion
AOR	Adjusted odds ratio
BC	British Columbia
EFA	Exploratory factor analysis
EU-OSHA	European Agency for Safety and Health at Work
ILO	International Labour Organization
IPC	Infection prevention and control
KMO	Kaiser-Meyer-Olkin statistic
OR	Odds ratio
OSH	Occupational safety and health
OSHA	United States' Occupational Safety and Health Administration
PA-MRFA	Parallel analysis - minimum rank factor analysis
PPE	Personal protective equipment
UCD	University College Dublin

Ethics approval and consent to participate

Informed consent to participate in this study was obtained from all participants. All participants were 18 years of age or older. Informed consent for publication of potentially identifying information in an online open-access publication was obtained from all subjects. All methods were performed in accordance with the guidelines and regulations outlined in the Declaration of Helsinki. This study was approved for exemption from full ethical review by the University College Dublin Human Research Ethics Committee – [Sciences (HREC-LS)]. Research Ethics Exemption Reference Number (REERN): LS-E-21-138-Perrotta.

Consent for publication

Not applicable

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Competing interests

The authors declare that they have no competing interests.

Funding

This research was funded by Science Foundation Ireland (grant number 20/COV/8539).

Authors’ contributions

CI: conceptualization, methodology, survey design and translation, data collection, validation, formal analysis, data curation, writing – original draft, visualization CP: conceptualization, methodology, resources, survey design and translation, data collection, writing – review & editing, supervision, project administration, funding acquisition YC: survey design and translation, data collection, writing – review & editing CB: writing – review & editing, funding acquisition VD, MA: survey design, data collection NR: survey design and translation, data collection MR: survey design, writing – review & editing AD: writing – review & editing. All authors have read and agreed to the published version of the manuscript.

Acknowledgements

This study was supported by the Science Foundation Ireland (Grant 20/COV/8539). Survey distribution was made possible thanks to partners at the UCD Centre for Safety and Health at Work, the BC General Employees Union, and EU-OSHA. We would like to thank Dr. Shiraz Syed for his translation of the survey into Hindi and Carine Grenier for supporting the French translation of the survey and related materials.

Bi Q, Wu Y, Mei S, Ye C, Zou X, Zhang Z, et al. Epidemiology and transmission of COVID-19 in 391 cases and 1286 of their close contacts in Shenzhen, China: a retrospective cohort study. Lancet Infect Dis. 2020;20(8):911–9.
Morawska L, Milton DK. It Is Time to Address Airborne Transmission of Coronavirus Disease 2019 (COVID-19). Clin Infect Dis. 2020 Nov 1;71(9):2311–3.
Leclerc QJ, Fuller NM, Knight LE, Funk S, Knight GM. What settings have been linked to SARS-CoV-2 transmission clusters? Wellcome Open Res. 2020 Jun 5;5:83.
Greenhalgh T, Jimenez JL, Prather KA, Tufekci Z, Fisman D, Schooley R. Ten scientific reasons in support of airborne transmission of SARS-CoV-2. The Lancet. 2021 May 1;397(10285):1603–5.
Ingram C, Downey V, Roe M, Chen Y, Archibald M, Kallas K-A, et al. COVID-19 Prevention and Control Measures in Workplace Settings: A Rapid Review and Meta-Analysis. Int J Environ Res Public Health. 2021 Jan;18(15):7847.
International Labor Organization. Protecting the life and health of workers during the COVID-19 pandemic: Overview of national legislative and policy responses, January 2021. Geneva: ILO; 2021.
European Centre for Disease Prevention and Control. Rapid Risk Assessment: COVID-19 outbreaks in long-term care facilities in the EU/EEA in the context of current vaccination coverage, 26 July 2021. Stockholm:ECDC; 2021.
Fegan C. More than one-third of Covid-19 cases in Ireland linked to workplaces. Independent [Internet]. 2021 August 16 [cited 2021 Aug 16]; Available from: https://www.independent.ie/irish-news/health/more-than-one-third-of-covid-outbreaks-linked-to-workplaces-40754437.html
United Food and Commercial Workers. UFCW: OSHA COVID Workplace Safety Standard Fails to Protect Frontline Grocery and Meatpacking Workers Still At Risk From Pandemic [Internet]. Washington, D.C: UFCW; 2021 [updated 14 June 2021; cited 2021 Aug 16]. Available from: https://www.ufcw.org/press-releases/ufcw-osha-covid-workplace-safety-standard-fails-to-protect-frontline-grocery-and-meatpacking-workers-still-at-risk-from-pandemic/
Occupational Safety and Health Administration. Subpart U—COVID-19 Healthcare Emergency Temporary Standard. Washington, D.C.: OSHA, 2021.
Johns Hopkins Coronavirus Resource Center. COVID-19 Map [Internet]. Baltimore: Johns Hopkins Coronavirus Resource Center; 2020 [updated 2021 October 05; cited 2021 August 16]. Available from: https://coronavirus.jhu.edu/map.html
Centers for Disease Control and Prevention. When You've Been Fully Vaccinated [Internet]. Atlanta: CDC; 2020 [updated 2021 September 16; cited 2021 Jun 15]. Available from: https://www.cdc.gov/coronavirus/2019-ncov/vaccines/fully-vaccinated.html
Hsu A. Federal COVID Workplace Safety Rules Are Here. But Only For Health Care Workers. National Public Radio [Internet]. 2021 June 10 [cited 2021 Jun 15]. The Coronavirus Crisis. Available from: https://www.npr.org/2021/06/10/1005036698/federal-covid-workplace-safety-rules-finally-here-but-only-for-health-care-worke
European Union. Re-open EU [Internet]. Brussels: EU; 2021 [updated 2021 Oct 05; cited 2021 Aug 16]. Available from: https://reopen.europa.eu
Smith K, Ramsay C. State of local emergency declared in Regional Municipality of Wood Buffalo amid high COVID-19 cases. Global News [Internet]. 2021 April 26 [cited 2021 Aug 12]; Health. Available from: https://globalnews.ca/news/7800424/state-of-local-emergency-to-be-declared-in-regional-municipality-of-wood-buffalo/
Purnell N. India’s Covid-19 Crisis Tests the World’s Back Offices. Wall Street Journal [Internet]. 2021 May 22 [cited 2021 Sep 28]; Available from: https://www.wsj.com/articles/indias-covid-19-crisis-tests-the-worlds-back-offices-11621656009
ILO. Country policy responses [Internet]. Geneva, ILO; 2021 [updated 2021 Oct 05; cited 2021 Aug 16]. Available from: https://www.ilo.org/global/topics/coronavirus/regional-country/country-responses/lang--en/index.htm
Ananda-Rajah M, Veness B, Berkovic D, Parker C, Kelly G, Ayton D. Hearing the voices of Australian healthcare workers during the COVID-19 pandemic. BMJ Lead [Internet]. 2021 Mar 1 [cited 2021 Sep 14];5(1). Available from: https://bmjleader.bmj.com/content/5/1/31
McRobert CJ, Hill JC, Smale T, Hay EM, Van der Windt DA. A multi-modal recruitment strategy using social media and internet-mediated methods to recruit a multidisciplinary, international sample of clinicians to an online research study. PLoS ONE. 2018;13(7).
Ball HL. Conducting Online Surveys. J Hum Lact. 2019 Aug 1;35(3):413–7.
Wejnert C, Heckathorn DD. Web-Based Network Sampling: Efficiency and Efficacy of Respondent-Driven Sampling for Online Research. Sociol Methods Res. 2008 Aug 1;37(1):105–34.
Hlatshwako TG, Shah SJ, Kosana P, Adebayo E, Hendriks J, Larsson EC, et al. Online health survey research during COVID-19. Lancet Digit Health. 2021 Feb 1;3(2):e76–7.
Alessi E. Conducting an Internet-based Survey: Benefits, Pitfalls, and Lessons Learned. Soc Work Res. 2010 Jun 1;34:122–8.
Souza AC de, Alexandre NMC, Guirardello E de B. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol E Serviços Saúde. 2017 Sep;26:649–59.
Baglin J. Improving Your Exploratory Factor Analysis for Ordinal Data: A Demonstration Using FACTOR. PARE. 2014; 19 (5):1–15.
Timmerman ME, Lorenzo-Seva U. Dimensionality assessment of ordered polytomous items with parallel analysis. Psychol Methods. 2011 Jun;16(2):209–20.
Tabachnick BG, Fidell LS, Ullman JB. Using multivariate statistics. New York: Pearson; 2019.
Streiner DL. Starting at the Beginning: An Introduction to Coefficient Alpha and Internal Consistency. J Pers Assess. 2003 Feb;80(1):99–103.
Ursachi G, Horodnic IA, Zait A. How Reliable are Measurement Scales? External Factors with Indirect Influence on Reliability Estimators. Procedia Econ Finance. 2015 Jan 1;20:679–86.
Watkins C, Daniels L, Jack C, Dickinson H, van den Broek M. Accuracy of a single question in screening for depression in a cohort of patients after stroke: comparative study. BMJ. 2001 Nov 17;323(7322):1159.
Polit DF. Assessing measurement in health: Beyond reliability and validity. Int J Nurs Stud. 2015 Nov;52(11):1746–53.
World Health Organization, ILO. Policy Brief: Preventing and mitigating COVID-19 at work. Geneva: WHO, ILO; 2021.
Department of Health and Human Services. Strategies to reduce COVID-19 transmission at the Smithfield Sioux Falls Pork Plant. Sioux Falls: CDC; 2020.
Silverberg SL, Puchalski Ritchie LM, Gobat N, Murthy S. COVID-19 infection prevention and control procedures and institutional trust: Perceptions of Canadian intensive care and emergency department nurses. Can J Anesth Can Anesth. 2021 Aug 1;68(8):1165–75.
Harvard Medical School. Coping with face mask discomfort [Internet]. Boston: Harvard Health; 2020 [updated 2020 Dec 01; cited 2021 Aug 10]. Available from: https://www.health.harvard.edu/staying-healthy/coping-with-face-mask-discomfort
Iftekhar EN, Priesemann V, Balling R, Bauer S, Beutels P, Valdez AC, et al. A look into the future of the COVID-19 pandemic in Europe: an expert consultation. Lancet Reg Health – Eur [Internet]. 2021 Jul 29 [cited 2021 Aug 16];0(0). Available from: https://www.thelancet.com/journals/lanepe/article/PIIS2666-7762(21)00162-9/abstract
Central Statistics Office. Social Impact of COVID-19 on Women and Men [Internet]. Dublin: CSO; 2020 [updated 2020 May 19; cited 2021 Sep 14]. Available from: https://www.cso.ie/en/releasesandpublications/er/sic19wm/socialimpactofcovid-19onwomenandmenapril2020/
European Commission. COVID-19 pandemic is a major challenge for gender equality [Internet]. Brussels: EU; 2021 [updated 2021 March 05; cited 2021 Sep 14]. Available from: https://ec.europa.eu/commission/presscorner/detail/en/IP_21_1011
MacDonald NE. Vaccine hesitancy: Definition, scope and determinants. Vaccine. 2015 Aug 14;33(34):4161–4.
Dzieciolowska S, Hamel D, Gadio S, Dionne M, Gagnon D, Robitaille L, et al. Covid-19 vaccine acceptance, hesitancy, and refusal among Canadian healthcare workers: A multicenter survey. Am J Infect Control. 2021 Sep 1;49(9):1152–7.
Murphy J, Vallières F, Bentall RP, Shevlin M, McBride O, Hartman TK, et al. Psychological characteristics associated with COVID-19 vaccine hesitancy and resistance in Ireland and the United Kingdom. Nat Commun. 2021 Jan 4;12(1):29.
Ruijs WLM, Hautvast JLA, Kerrar S, van der Velden K, Hulscher MEJL. The role of religious leaders in promoting acceptance of vaccination within a minority group: a qualitative study. BMC Public Health. 2013 May 28;13:511.
Thornton L, Batterham PJ, Fassnacht DB, Kay-Lambkin F, Calear AL, Hunt S. Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interv. 2016 May 1;4:72–81.
Davies B, Kotter M. Lessons From Recruitment to an Internet-Based Survey for Degenerative Cervical Myelopathy: Comparison of Free and Fee-Based Methods. JMIR Res Protoc. 2018 Feb 5;7(2):e6567.

No competing interests reported.

Development and Validation of a Multi-lingual Online Questionnaire for Surveying the COVID-19 Prevention and Control Measures used in Global Workplaces

Status:

Journal Publication

Version 1

Abstract

Figures

Background

Methods