This cross-sectional, pilot study was conducted in three phases: 1) survey development, 2) pilot testing, and 3) psychometric property evaluation. Psychometric properties assessed as part of the study are defined in Figure 1. The methodological steps taken to validate them and finalize the survey are outlined in Figure 2.
Phase 1. Survey Development
Creating the survey
From February to April 2019, the researchers conducted an extensive review of relevant research literature and meta-analysis to investigate the effectiveness of measures to prevent and control COVID-19 outbreaks in global workplace settings. Results of this review identified that combinations of (1) basic preventive measures (i.e., masking, hand hygiene, and social distancing), (2) surveillance measures, (3) outbreak investigations and response, (4) environmental adjustments, and (5) education initiatives can effectively prevent workplace outbreaks (5). These categories became the framework for the development of a survey allowing researchers to measure the level of occupational COVID-19 protection available to workers from diverse geographic and occupational settings.
A preliminary questionnaire in English was designed on QualtricsXM (Provo, UT) online survey platform with questions pertaining to:
- Basic Demographics (age, gender, country of residence, education);
- Employment Demographics (workplace category, industry, size, current role);
- Basic preventive measures (hand hygiene, barriers, social distancing, masking, etc.);
- Environmental adjustments (temperature, ventilation, air quality monitoring);
- Surveillance (type of testing, access to testing, self-isolation requirements, syndromic surveillance);
- Contact tracing;
- Education and training; and
Questions were designed for actively working employees, managers, and occupational safety and health professionals from any country. Skip patterns were enabled based on participant characteristics. For example, managers and employees were given access to separate question tracks, and only participants that selected “Healthcare and Social Assistance” as their occupational sector could access questions on medical PPE. As the survey was intended for internationally and occupationally diverse participants, survey questions were worded as generally and simply as possible.
Establishing content validity
The preliminary survey was independently reviewed by five multinational experts in Public Health and Occupational Safety and Health to ensure face validity (that the questions adequately met the study’s aims) and content validity (to ensure that questions were all relevant to the study’s aims). Once independent review was complete, experts convened for two, one-hour meetings with research team members to discuss findings. Survey items that did not reach consensus during the first meeting were removed, modified, or reworded by the researchers and presented again for expert review. During the second meeting, experts reached consensus on all survey items as well as on general format, language, and response options.
Prior to dissemination, the Qualtrics survey was pre-tested with 10 actively working individuals from Ireland and the United States. Researchers met one-one-one with construction (n=1), retail (n=2), education (n=3), manufacturing (n=1), and healthcare (n=3) employees to ensure adequate comprehension of the information sheet, survey items, and response options. Pre-testing data was reviewed, and the survey adapted based on participant feedback. Researchers added questions pertaining to a pandemic transition phase (i.e., measures that workers would like to see upheld/removed as vaccinations increase), and whether or not workers feel adequately protected from COVID-19 at work. Pre-testing allowed researchers to identify and correct errors in skip patterns.
Translating the survey
The survey instrument, once finalized in English, was translated into Spanish, French, Polish, Hindi, and Chinese. Members of the research team with native-level fluency in these languages conducted forward translation. Subsequent back-translation using Google TranslateTM allowed researchers to identify and fix any discrepancies. All translated versions of the survey were pre-tested with at least one actively working native speaker to ensure ease of comprehension of translated items.
Phase 2. Pilot Testing
Participants were recruited between 01 July and 01 August 2021 through non-probability convenience sampling techniques. Included participants were working individuals aged 18 or older with literacy in English, French, Spanish, Hindi, Polish, or Chinese. Participants who were not actively working or who were currently working full-time from home were not excluded from participating. Instead, they were directed to a shortened version of the questionnaire with questions pertaining to (1) basic demographics, (2) employment demographics, and (3) vaccination status. Because working from home is a safety measure in itself, the inclusion of said participants was considered relevant to the survey’s objectives.
Workers in countries where the research team had professional contacts and/or where appropriate survey translations were available were targeted in this pilot study. To ensure a geographically and occupationally diverse sample, researchers advertised the survey link through multiple recruitment channels in a manner similar to McRobert et al. (19):
- Formal recruitment channels: OSH organizations and national trade unions were contacted by the researchers and, if willing to collaborate, sent the survey link to all constituents via email;
- Social media groups: Researchers joined relevant occupational LinkedIn and Facebook groups and posted the survey link into those groups; and
- Personal social media: Researchers’ posted the link to their own social media profiles (Facebook, LinkedIn, Twitter, WhatsApp).
The multi-modal online participant recruitment strategy was approved by the University College Dublin Human Research Ethics Committee (LS-E-21-138-Perrotta) and followed recommended procedures for online surveys. By keeping careful records of where, when, and by whom participants were contacted, and the number of participants contacted (20), partnering with other organizations and encouraging peer-led snowball sampling (21), and using an open access, ethically approved, and user optimized instrument (22), the researchers limited potential for sample bias (20,23). Professional collaborators who agreed to advertise the survey included the European Agency for Safety and Health at Work (EU-OSHA), the University College Dublin Centre for Safety and Health at Work, and the British Columbia General Employees’ Union. The survey link was posted in 22 international, OSH-themed LinkedIn and Facebook groups accounting for 270,000 total group members.
To assess associations between participant characteristics and workplace protection, odds ratios (OR) and adjusted odds ratios (AOR) generated from logistic regression models were used to estimate the likelihood of feeling protected at work vs. feeling unprotected or unsure according to age, gender, country of residence, education level, occupation, and vaccination status. Stepwise model selection by Akaike information criterion (AIC) was performed to determine the best-fit multivariable model using R version 4.0.2 step() function (R Foundation for Statistical Computing, Vienna, Austria). Survey data used for logistic regression is provided in Additional File 1.
Phase 3. Psychometric Property Evaluation
Quantitative survey validation followed recommended guidelines for survey reliability and validity testing (24). All data were analysed in R. Note that content and cross-cultural validity were tested prior to data collection during the survey development phase. Survey reliability, criterion and construct validity were evaluated following data collection.
Prior to data collection, the researchers hypothesized that preventive constructs fell under seven IPC measures’ domains: basic preventive measures, environmental adjustments, surveillance, contact tracing, PPE, education and training, and vaccination. To test this, exploratory factor analysis (EFA) was conducted using R package ‘psych’ 2.1.6 to identify the underlying factor structure explaining the relationship between 44 measured variables (25). Questions relating to medical PPE and contact tracing were excluded from EFA due to large quantities of missing data. To determine the suitability of data for EFA, Pearson correlation matrices were verified for a statistically significant Bartlett’s test and a Kaiser-Meyer-Olkin (KMO) statistic above 0.60 (25). Scree plot inspections and parallel analysis based on minimum rank factor analysis (PA-MRFA) were conducted to determine the advised number of factor dimensions. The PA-MRFA method, which expresses model fit by reporting the overall percentage of common variance explained (25), has been recommended for assessing the number of common factors underlying polytomous variables (26). The Oblimin oblique rotation (non-orthogonal) method was used to aid factor interpretation, with rotated loadings above 0.32 considered acceptable (27). Data and codes used for EFA, and a description of coded response options are available in Additional Files 2-3.
Cronbach’s alpha coefficients and values of alphas if removed were calculated to measure internal consistency within the IPC domains identified through EFA. Values ≥ 0.6 indicated acceptable internal consistency – that is, that all questions within the domain measured the appropriate protective construct – a threshold that has been recommended in the early stages of research (28,29).
Criterion validity is the relation between the score of an instrument and, conventionally, another instrument that is widely accepted as a ‘gold standard’ (24). Because this was the first survey intended to measure the level of COVID-19 protection in a participants’ workplace, no accepted instrument existed for comparison. Instead, researchers asked participants to identify whether or not they felt protected in the workplace at the end of the survey, determining whether investigated protective measures related to participants’ own standards for adequate safety. This method followed a similar procedure to Watkins et al. who looked for an alternative to a long instrument to assess depression and tested a single question – Do you frequently feel sad or depressed (30)? This study sought to verify a long instrument to assess COVID-19 workplace safety by comparing it to a single question – Do you feel protected from COVID-19 at work?
To test for criterion validity, an overall COVID-19 IPC measures’ protective score was calculated out of 40 possible points and compared to the survey’s ‘gold standard’ question. Respondents received one point for each affirmative response to questions on basic preventive measures, environmental adjustments, testing and surveillance, education and training, PPE, contact tracing, vaccination status, and access to paid sick leave. Detailed score calculations are provided in Additional File 4. Protective scores were compared to whether or not a worker felt protected from COVID-19 at work using two-sample t-tests; the significance level was set at p < 0.05.
To understand the extent to which a set of variables represent the construct intended to be measured (i.e., construct validity), discriminant validity tests the hypothesis that a target-measurement is not improperly related to variables from which it should differ (31). As the survey aimed to measure the level of occupational COVID-19 protection available to workers, a higher number of IPC measures in place should not be associated with feeling unprotected at work and vice versa. To test this, respondents were divided into quartiles according to total protective measures scores. Odds ratios (OR) generated from univariable logistic regression were used to estimate the likelihood of feeling protected at work according to protective score quartile. Known-groups technique was applied to assess whether the survey instrument successfully captured expected differences between groups of survey respondents. Hypotheses were formulated prior to data collection and tested using two-sample t-tests (significance level: p < 0.05).