Protocol Availability and Ethical Approval
The study protocol was approved by the local Brazilian Institutional Review Board (IRB) (Comitê de Ética em Pesquisa da Faculdade de Medicina, Universidade Federal de Pelotas, Brazil/number 10227419.2.0000.5318), and the full research protocol is available on the Open Science Framework platform (https://osf.io/2ut5v/).
This study was designed as a randomized, 1:1, superiority, parallel-group, blinded (for assessors) and controlled by gender and career stage trial, comparing the researchers’ assessment for the same CV with a male or female gender, using a selection process for a post-doctoral position in Dentistry at a southern Brazilian University as a proxy. The study's expositions were the gender of the applicant at two levels (male and female) and the career stage of the applicant at two levels (early-stage or later stage careers). The primary outcome was evaluators' assessments in each of the four categories evaluated (scientific contribution, leadership potential, ability to work in groups, and international experience) according to CV’s gender and career stage.
This study was reported based on the CONSORT 2010 Statement and its extension for multi-arm randomized trials.7,8
Eligible participants (i.e., ad-hoc assessors) were level 1 and 2 research fellows from the Brazilian National Council for Scientific and Technological Development of the year 2020 in dentistry. Because of our university community's potential knowledge about our trial, we decided to exclude our research fellows from the sample to avoid possible contamination bias.
The sample size estimation was based on the results of a previous study.2 and measures of clinical relevance. We assumed a maximum type 1 error of 0.05, a power of 0.90, and an effect size of -0.81 (mean difference between groups at the final grade) with a standard deviation of 1.1. We obtained a sample size of 78 researchers. Considering the average response rate of 10% in questionnaires9, and a non-response rate of 90%, we assessed eligibility for all 211 research fellows in dentistry. We randomized all 117 researchers who met the inclusion criteria and agreed to participate.
Randomization and blinding
Researchers were randomly assigned to receive a female or male CV with a 1:1 allocation per a computer-generated randomization system stratified by career stage (early-career or non-early career) using permuted blocks of random sizes. The list of random numbers was made on a website (www.sealedenvelope.com). The concealment of participants’ allocation was warranted by a researcher not involved in the study, and another researcher allocated each participant following the allocation sequence. Each researcher received only one CV.
The researchers were not aware of the study. They were invited to act as external peer reviewers in a selection process for a supposed post-doctoral position at a southern Brazilian university.
Each of the researchers selected according to the eligibility criteria received an e-mail (Appendix 1) with an invitation to act as an ad-hoc reviewer in a supposed post-doctoral fellowship. In case of acceptance, each researcher received a second e-mail (Appendix 2) containing information about the process, which s/he was invited to be part of the evaluation process. Along with the information process, this e-mail had one of the four CV possibilities to be evaluated. The options were: early-career female (Appendix 3), early-career male (Appendix 4), non-early career female (Appendix 5), non-early career male (Appendix 6). For more information, see Table 1. This e-mail also contained a document with a simulated call for application (Appendix 7) to give credibility to the process.
Table 1. CV possibilities to be evaluated
Male + Early Career
Male + Not Early Career
Female + Early Career
Female + Not Early Career
*CVs from the same career stage were strictly the same, except for gender identification
The CV considered as “early career” contained information by an applicant who just concluded his Ph.D. and has 12 papers published, compared to the CV considered as “non-early career applicant.” It contained information about an applicant who will have a previous post-doctorate and more than 20 papers published. The idea of different CVs degrees was to assess if gender bias occurs more at the beginning of the career or when the career is more consolidated.
Each researcher received a CV of an applicant (gender and career stage selected according to the randomization) and was required to rate each topic from 0 to 10 (0 being insufficient and 10 very sufficient) on a visual analog scale. The topics were scientific contribution, leadership potential, ability to work in groups, and international experience. To allow the blinding of the evaluators and the equivalence of the male and female CVs and profiles, information on the full name and the publication list was blinded, as well as any external reference that could be cross-checked online, such as Researcher ID, ORCID ID, social media profiles, grant numbers, etc. The researchers were also not aware that they were participating in a study. However, when they sent the CV assessment, they received an e-mail with a questionnaire (Appendix 8) containing information about the study and requesting authorization to use the previously submitted data. Researchers were also asked whether, at any time, they have ever suspected the veracity of the process for selection of a fellow post-doctoral researcher.
The same researcher (MCF) sent all e-mails containing invites, the explanation of the study, and the Free Prior Informed consent (FPIC) from an institutional e-mail created for this purpose.
The primary outcome was the final grade given by evaluators according to CV’s gender and career stage. As a secondary outcome, we evaluate the each one of the four categories evaluated (scientific contribution, leadership potential, ability to work in groups, and international experience) and the grades in each category according to the gender of CVs’ and evaluators’ gender. Each category could receive grades from 0 (lowest score) to 10 (excellent score), being the final grade an arithmetic mean of the four grades given in each item. Grades could also contain decimal numbers (e.g., 9,6).
Descriptive analyzes were used to summarize evaluators' characteristics. Continuous variables were described as mean and standard deviation or interquartile range (IQR). Categorical variables were expressed as point estimates and 95% confidence intervals. Gross binary associations between exposures and outcomes were conducted by X2 tests within an alpha of 0.05 for significance. Step forward robust variances Poisson regressions with log links were undertaken to estimate associations of exposure variables of interest (gender and career stage) to dependent variables - scientific contribution, leadership potential, ability to work in groups, and international insertion), both adjusted for CV gender and career stage and non-adjusted. All analyzes were performed using the software SPSS statistics 25 (IBM, Nova York, USA), and an alpha level of 0.05 was set for inferential analyzes.