Image Generation
Dall-E, a text-to-image tool, was used to generate synthetic images, corresponding to the terms listed in Supplementary Table 8. A diagram illustrating the hierarchical labeling process can be found in Fig. 1A. For each term, 4 individual images were generated, that were extracted as one composite, as shown in Fig. 1B. A total of 90 composites consisting of 360 images were generated between December 2022 and July 2023. The medical professions analyzed include “doctor,” “physician,” “internist,” “surgeon,” and “nurse.” For each set of images, different race and sex labels were added to further refine the DALL-E query. The sex labels selected were “male” and “female.” The race labels selected were “Black,” “White,” “American Indian,” “Pacific Islander,” and “Asian,” based on U.S census bureau recommendations12.
Quality and Realisticness Assessment of Images
We evaluated images for quality, realisticness, and presence of clipart. For quality, we assessed if the image depicted a full vs partial face. For realisticness, we assessed distortion of facial features in the images, such as blurriness, uneven features (such as eye colors, eye shapes), and color distortion. We also recorded the number of individuals (from 0–4) that were clipart for each Dall-E generated image. (See Supplement).
Qualitative Assessment of Diversity
We evaluated sex, race, and age diversity for each composite consisting of 4 images. The composites were rated on a scale from 1–5 (for sex and race diversity) and 1–3 (for age diversity) based on consensus scoring (AA, AA, GG).
-
Sex Diversity - a measure of the apparent representation of female and male sexes
-
Race Diversity - a measure of the apparent representation of different races
-
Age Diversity - a measure of the apparent representation of youthfulness
A full explanation of our scale for each of these following factors can be found in the Supplement and Supplementary Tables 1–3. Additionally, we provide an example rating of the term “Doctor” in Fig. 1B.
ML-Driven Image Labeling
Google Vision is a tool that utilizes computer vision algorithms to assign labels and identify objects within images. Here, we used the Google Vision API to assign the top 10 labels for each image. Each label is associated with a confidence score, which we then normalized using the following formula:
Normalized score = \(\frac{score - min}{max - min}\)
where min is the minimum score amongst all labels, and max is the maximum score amongst all labels.
Integrated Assessment of Labels and Categories
We categorized each label into a category, with a set of 12 unique categories, as defined in the below Table 1. The full set of labels, along with its normalized score and category, can be found at Supplementary Table 9.
Table 1
Category | Definition | Top Terms |
Facial Expression | Terms related to emotions and the act of expressing such emotions. | 1. Smile 2. Facial Expression 3. Happy |
Clothing (Formal) | Clothing that is formal (typically associated with a professional setting) but is not Medical Clothing. | 1. Collar 2. Dress Shirt 3. Workwear |
Clothing (General) | Clothing that is not formal nor medical clothing (includes clothing for everyday use). | 1. Sleeve 2. Outerwear 3. Clothing |
Medical Clothing | Clothing specifically worn by professionals working in healthcare. | 1. White Coat 2. Scrubs 3. Protective Personal Equipment |
Medicine | All medical terms that were not Medical Tools or Medical Clothing. | 1. Service 2. Health Care Provider 3. White-Collar Worker |
Medical Tools | Instruments used by medical professionals that detect, prevent, and treat illnesses. | 1. Stethescope 2. Scrubs |
Fashion Occupation | Terms related to the field of fashion. | 1. Fashion Design 2. Fashion Accessory 3. Jewellery |
Color | Colors, typically characterizing colors in the image. | 1. Electric blue 2. White 3. Azure |
Body Part | Internal and external components of the human body excluding facial features. | 1. Neck 2. Shoulder 3. Joint |
Miscellaneous | Any labels that did not belong in the other categories. | 1. Event 2. Product 3. Photograph |
Facial Feature | Body parts that reside on the face. | 1. Jaw 2. Eyelash 3. Eyebrow |
Actions | Activity or movement done by a human. | 1. Gesture 2. Standing 3. Animation |
Hierarchical Clustering and Heatmaps
In order to identify the overall similarity of images from the same cohort and major difference between two cohorts, we performed hierarchical clustering. Ward’s clustering method was used. To measure the difference between two images, Euclidean distance was used with the following formula:
d(p, q) = \(\sqrt{{\sum }_{i=1}^{n}{{(q}_{i}-{p}_{i})}^{2}}\)
where q represents the sum of normalized scores for all labels for a specific category in image A and p represents the sum of normalized scores for all labels for the same category in image B. n represents the total number of categories, which is 12.
Size of cluster is defined by the number of images of one cohort that are consecutive within a cluster.
Heatmaps were also generated to depict the distribution of categories in different cohorts. We calculated the proportion of labels belonging to each category, and compared these proportions across the different sexes and races in distinct heatmaps.
Statistical Analysis
A one-sided z-proportion test was used to compare proportions and percentages. A p-value < 0.05 was used to indicate statistical significance. A Chi-Square test was used to compare entire distributions of categories.