Content and Accessibility of United States Medical School Websites During COVID-19 and the Online Interview Season

Abstract


Abstract Background
Most medical school applicants use the internet as a source of information when applying for medical school. Previous analyses have evaluated residency and fellowship websites, however, an in-depth analysis regarding medical school websites is lacking.

Methods
We evaluated 192 United States medical school websites for presence or absence of 39 items relevant to medical school applicants. Items fell into 7 general categories: curriculum, research, demographics of incoming class, admissions information, faculty, nancial aid, and social.

Results
Of the 192 websites evaluated (152 allopathic and 40 osteopathic schools), websites contained a mean of 23 items (59%) with a standard deviation of 4.2 items.

Conclusion
Our study examining United States Medical School websites revealed a lack of online information for medical school applicants. As medical school interviews transition to being online during the COVID-19 pandemic, the importance of the medical school website to applicants becomes increasingly crucial. The information contained in our study could be used to improve accessibility and information on medical school websites, which could help both applicants and medical schools themselves. website for program information.
[2] In May 2020, a group of national medical education organizations named the Coalition for Physician Accountability made speci c recommendations to medical schools and residencies due to the COVID-19 pandemic. These recommendations are predicted to affect at least the 2020 through 2021 application cycle, and will signi cantly limit opportunities for personal interactions between medical school applicants and medical schools. [3] Accordingly, website development, content, and accessibility is increasingly important for medical schools. Prior studies have evaluated these factors among residency and fellowship programs, resulting in various recommendations for areas of improvement among their respective program websites to both help applicants and increase recruitment. [4][5][6][7][8][9][10] Comprehensive websites can help applicants make informed decisions, providing them with access to information they deem important. For the application cycle of 2019 to 2020, medical school applicants applied to an average of 17 medical schools despite according to data recently released by the Association of American Medical colleges.
[1] Medical school applications and interviews are costly for both applicants and programs. Providing applicants with more information to guide decisions regarding which programs to apply and interview at stands to bene t both parties, especially if it results in a better matching of applicants likely to t a school. Medical schools also stand to bene t from better matching of applicants, as many institutions ll residency positions with recent graduates of their own programs.
The astounding number of applications could be due to the increasingly competitive nature of medical school or stress and concern for medical school acceptance. These factors may be accentuated during the COVID-19 pandemic as the Medical College Admissions Test has become delayed, and many service and shadowing opportunities have been cancelled. [11] In addition, the number of applications could be augmented due to lack of availability of information on medical school websites, prompting students to apply to more programs than they are realistically interested in.
Information sources such as the MSAR online database and the American Osteopathic Association (AOA) website were designed to assist medical school applicants while applying for medical school. [12,13] Given the growing reliance on technology to do personal research on different medical schools, the need for prospective medical students to nd robust, consistent information online will continue to increase.
The main purpose of our study is to provide an in-depth analysis of medical school website content and to assess the functionality of MSAR and AOA for prospective medical school applicants. To our knowledge, this study is the rst to do an in-depth analysis of medical school websites in the United States.

Methods
The methods of our study were adapted from a well-conceived study examining otolaryngology residency website content. [5] Our study was exempt from Institutional Review Board (IRB) approval because it involves publicly available information. A list of 192 medical school names and website links were obtained from Medical School and Admissions Requirements (MSAR) online database and the American Osteopathic Association (AOA) website in May 2020. We evaluated both information sources for functionality by determining whether the link provided on MSAR or AOA led directly to the medical school homepage or required multiple clicks to get to the medical school homepage. When a link to a program was not available on MSAR or AOA website, we performed a Google search to nd the program website. Medical schools without a functional website, or a website that could not be found, were excluded. The data were collected by four authors (JW, JS, DM, JR) between May 2020 and June 2020. Data gatherers searched the websites of these programs for 39 items listed in Table 1. These items were later divided into 7 categories for further analysis: curriculum, research, demographics of incoming class, admissions information, faculty, nancial aid, and social. The items and categories included in our study are based off variables many applicants deem as necessary or desirable information, as well as variables we believe are important to medical school applicants. [14] We also included items from a variety of studies examining the quality of residency website content, and based our 7 categories off these studies. [4-6, 9, 10] Some of these factors on medical school websites could be more important to applicants than other factors, as suggested by studies examining how medical students choose a residency program. [15] However, we controlled for this by examining a large number of items (39) on each website and based these items on a variety of studies, as previously described. As the data contained in residency websites can be subjective, we created a standardized process to evaluate the websites, similar to the previous studies in other specialties. [4][5][6][7][8] First, we only searched for the presence or absence of items, with no attempt made to grade the quality or accuracy of the content. Second, any information that was not directly listed on medical school website was excluded, such as links to external materials or websites, which usually contained general, non-speci c information rather than medical school-speci c information. Lastly, data were gathered independently by four authors for the same ten programs and compared for agreement. All authors went through all items again together, noting where disagreement occurred. Ambiguity in exclusion and inclusion criteria was resolved and adjusted accordingly upon agreement by all authors. After this instruction, data collectors independently gathered the data for all websites (JW, JS, DM, JR). For the remaining websites, disagreement was resolved by a fth author (LM). We performed a descriptive analysis of the data, including means and standard deviations. Additionally, a sub-analysis was performed to determine whether a difference exists in website quality and accessibility among top 25 medical schools and the other 167 schools in our study. We referred to the 2021 data from the US News and World Report website for best medical schools for research, which attempts to rank medical schools based on a variety of criteria. [16] Excel was used for statistical analysis.

Results
Of the 192 websites evaluated (152 allopathic and 40 osteopathic schools), websites contained a mean of 23 items (59%) with a standard deviation of 4.2 items. We found a statistically signi cant difference between allopathic and osteopathic programs (p < 0.05) for 6 of the 39 variables included in our study, which were USMLE Step 1 pass rate, facility description, research requirement information listed, scholarship opportunities listed, waitlist information provided, and early decision information provided. The variables contained least frequently among all medical schools were average nancial aid amount (3.6%), average United States Medical Licensing Examination (USMLE) Step 1 score (6.8%), USMLE Step 1 pass rate (15.6%), and US News and World Report ranking (15.6%). The variables contained most frequently were yearly overview (96.9%), required pre-medical courses (96.4%), admissions o ce contact information (94.8%), and information on how to apply for nancial aid (93.2%) [ Table 1].
The categories with the most amount of information across both allopathic and osteopathic schools were the social and nancial aid categories, with 81% and 69% of websites containing this information, respectively. The categories with the least number of variables was the curriculum and research categories at 50% and 54% respectively. Allopathic websites were more likely to contain variables relating to curriculum, research, demographics, admissions information, and faculty, while osteopathic websites were more likely to contain information in the nancial aid and social categories [ Figure 1].
Medical school websites from US News and World Report top 25 schools contained more of the 39 variables than schools from non-top 25 programs, with statistical signi cance demonstrated with variables such as research requirement, average nancial aid, in-state vs out-of-state preference listed, and US News and World Report ranking listed [ Table 2]. Medical school websites from top 25 schools had more variables listed on average in each of the seven general categories when compared to the other nontop 25 programs [ Figure 2]. Lastly, in-terms of website accessibility, 89% of medical school program listings on the MSAR database or the AOA website provided direct links, 10% provided absent or non-functional links, and 1% provided indirect links.

Discussion
As medical school applicants apply to, interview at, and ultimately decide which institution to attend, careful planning and research is essential. The internet is easily accessible and multiple studies have shown the importance of websites in recruitment for residency, 5,8,9 which likely applies to students applying for medical school as well. This will likely be of growing importance in the wake of the COVID-19 pandemic, as in-person methods of evaluating prospective institutions are being replaced with virtual means. Medical schools spend a substantial time and effort recruiting competitive and diverse students throughout the year with advertisements, information sessions, carefully planned interview days, secondlook weekends, phone calls, nancial aid offers, and more. [19] While these efforts will continue to be important, maintaining a medical school website with adequate information and quality is paramount, particularly for today's prospective student. For example, a survey of medical school students applying for residency found that 41% of applicants decided not to apply to at least one program solely based on the quality of its residency website, and 78% of applicants claimed information provided in the residency program website in uenced their decision to apply to a particular program. [20] A recent study examining prospective students' medical school preferences reported the most important factors for medical school choice were academic prestige, location, and the "intangibles," such as "gut feelings" and personal interactions. [21] According to our study, medical schools rarely included US News and World Report rankings on their websites, average USMLE Step 1 score, or USMLE Step 1 pass/fail rate. These factors, which are associated with the prestige of the school, should be incorporated onto medical school websites. With regards to location, medical schools provided descriptions of the location of the medical school nearly 70% of the time. This becomes increasingly important with online interviews as many applicants may not be able to see the area for themselves. Another area for improvement for medical school websites could be providing state-by-state demographic information, which was only listed 19% of the time. However, the "intangibles," such as "gut feelings" and personal interactions were student wellness resources listed, clubs and interest groups, and social media links which were listed 90%, 81% and 84% of the time. This shows medical schools are likely aware of the "intangibles" and attempt to address them with more personal and social content.
With the rise of social media for recreational and professional purposes, integrating social media effectively and e ciently could help medical schools recruit desired applicants and help applicants learn more about different schools. We found 84% of medical schools' websites contained links to a form of social media representing their program, but 16% of programs did not have a directly accessible social media page for their program. This suggests an area for improvement. Having a social media site available for applicants could prove useful to students and programs, as a study done involving nearly 1000 medical students applying for residency showed that 68% of students reported using social media to learn about programs and 10% reported that the information found in the social media pages in uenced their decisions on where to apply. [22] Similarly, a survey of medical students applying for residency suggested social media as an e cient method for highlighting social activities to improve recruitment. [23] Information on current enrolled medical students, such as class demographics and state-by-state data, may be the only exposure of such applicants to the unique personalities and backgrounds of students in the program before deciding to apply to a program. Class demographics and state-by-state data were only listed on 61% and 19% of medical school websites, respectively. Medical schools may bene t from improving these sections of their websites. We also found medical school websites do not have much information pertaining to individual students. However, many residency websites often include photos of the class, photos of each individual student, and small, personal descriptions of each resident. This is an area for medical school websites to give more personalization to their program. Of course, maintaining appropriate con dentiality of the students should be considered.
Additionally, we found curriculum and research were not adequately addressed on medical school websites. For example, attendance policy was only listed on 27% of websites, grading policies were listed on 34% of websites, and research requirement information was listed on 25% of websites. As medical education becomes more personalized to the needs of individual students, understanding the individual needs of students, details such as lecture attendance policy, grading policies, and research requirements are helpful details that could help applicants choose which programs to apply to based on their unique learning style. [24][25][26] These details are essential as applicants choose to which programs they will apply. Also, while many of these topics are discussed during interviews, these details can be forgotten or unclear, and a more robust website would be useful in addressing these important questions when medical students are deciding which school to attend. Including more information on curriculum and research could help medical students decide which programs to apply to and attend and help programs recruit students who are a better t for the curriculum of the school. Recent articles have suggested that residency programs expand the amount of information for applicants during the COVID-19 interview cycle, and perhaps the same should apply to medical schools. [27] While some aspects of medical school websites are lacking, we found medical school websites are very accessible through the MSAR database and the AOA website. Of the 192 medical schools examined in our study, 89% of medical school program listings in AOA or MSAR provided direct links, while only 10% were absent or non-functional, and 1% were indirect links. However, the number of multiple-step, absent, and non-functional links could still be improved.
The top 25 schools in the United States according to United States News and World Report 2021 had more study variables listed on their websites than non-top 25 schools, such as information on the research requirement and average nancial aid. In addition, allopathic websites were more likely to contain variables relating to curriculum, research, demographics, admissions information, and faculty, while osteopathic websites were more likely to contain information in the nancial aid and social categories. Future investigation should determine why these differences exist, and whether these differences affect prospective student recruitment.
Future research should aim to determine how the COVID-19 pandemic and lack of online information on medical school websites could be affecting the number of applications submitted per applicant.
Limitations of our study include the subjective nature of analyzing medical school websites. However, we feel our method of data collection was standardized su ciently to control for ambiguity. Another limitation was the lack of established standardized criteria for evaluating websites. Every item that could be important to a medical school applicant was not analyzed. As a solution, we included a variety of items and developed search criteria based on studies evaluating residency website content and the recent experience of the authors of our study. Lastly, only including items listed directly on the medical school website rather than on external links could underestimate the presence of items on websites in our study.
However, this was an important factor to determine the accessibility of information and user-friendly status of the websites. Our study does not address accuracy or quality of information contained on websites. Notwithstanding these limitations, we believe our analysis provides valuable insight for medical school directors, website developers, and medical school applicants. Future areas of study could include an in-depth analysis of social media use among medical schools, how website quality affects the number of medical school applications, and why differences in website content exist between different programs, such as allopathic, osteopathic, and US News and World Report top 25 schools.

Conclusion
The 2020-21 residency application cycle poses a new challenge for applicants and programs. A study examining residency interview costs discovered four in ten students decline interviews for nancial reasons. [28] As most interviews are being conducted on a virtual platform, interview costs are less of an issue for applicants, which could drive up the number of programs students apply to and consider. Recent reports estimate a 14% increase in the number of medical school applications to AAMC compared to last year. [29] The sheer number of applications could overwhelm medical schools and increase competition for an already limited number of interview slots. As economic barriers of travel costs are brought down by the virtual interview mandate, students have the opportunity to apply to and attend more interviews. However, interview slots could potentially be taken up by a smaller pool of applicants. Now more than ever, a robust source of information available to applicants on medical school websites serves to bene t both applicant and program alike to prevent overwhelming medical schools with applications and interviewees. [30] Medical school website quality is important to medical school applicants, and our study identi es several areas where programs could focus efforts for website renovation, including improving the integration of social media and improving information on the curriculum and policies of the school. The results from our study can be used to improve medical school websites to the bene t of applicants and medical schools.

Declarations
Ethics approval and consent to participate: IRB approval was not required for our study based on Mayo Clinic IRB guidelines Consent for Publication: Not applicable Availability of data and materials: Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study Competing interests: None Funding: none Authors contributions: All authors contributed to the creation and implementation of the project design and implementation. JP and LM were responsible for statistical analysis, tables, and gures. All authors were responsible for critical revision and contribution to the nal draft of this manuscript. All authors read and approved this nal manuscript.