AJSP graduates and trainees outperformed their control group peers in most of our outcome measures of interest:
We incorporated membership as a proxy variable to determine trainees’ commitment to careers in child and adolescent psychiatry, particularly after graduation. There is no other streamlined way to determine career trajectory. The three-fold higher AACAP membership rate in the AJSP group indicates its participants are and remain committed to working in the field.
This is no trivial point, given that many promising would-be investigators in child psychiatry end up doing research in other fields given the long training duration and lag time necessary for subspecialized work with children and adolescents. Our findings suggest, albeit indirectly, that graduates of the AJSP are more likely to become active contributors not just to research in psychiatry, but specifically within the area of child and adolescent psychiatry. The high success rate in applying for AACAP awards, the larger number of awards received, and the shorter duration to first award all suggest that the AJSP has been effective in ‘socializing’ its participants from early on in their training into the work of, and opportunities provided by the Academy.
We tracked ABPN certification as a means of determining trainee’s clinical competency. AJSP participants were more likely to get board-certified in CAP compared to control peers, and despite the emphasis on research in the residency program, AJSP trainees did not take significantly longer to receive their initial board certifications compared to their control group peers. It should be noted that board certification may not be pursued by some graduates - particularly those opting for translational research careers, or those going to academic positions in which there may be less time pressure to obtain certification. Despite these caveats, AJSP graduates were still more likely to become board certified in CAP.
The majority of participants in both groups published peer-reviewed work indexed by the National Library of Medicine. However, participants in the AJSP were considerably more prolific, as reflected by a greater number of published and first-authored articles, and a shorter time to first publication. The fact that mean and highest impact factor did not differ between the groups indicates that participants in the AJSP were not simply publishing more articles but publishing in comparable quality venues. The h-index of AJSP participants was higher, suggesting more heavily cited and visible scientific output. This finding is notable in that the h- index, like the impact factor, has shortcomings that include the time required to accrue citations.28 Indeed, cumulative citations did not differ between the two groups, showing the long lag time needed for citation accrual.
A commonly used ultimate metric of scientific independence is receipt of NIH funding, particularly R-series grants. Participants in the AJSP had higher success rates in obtaining K-, but not R-series funding. The average dollar amount did not differ between groups. Given the low number of R-series awards (five) among this sample of 90 participants, and the long lag time to obtain a first R (an average of 10.6 ± 1.5 years from program enrollment) it behooves educators and funding agencies contemplating disbursement of funds to consider alternative metrics of scientific productivity in the shorter term.
Active components of the AJSP
We would have been delighted to take into the AJSP any one of the applicants in our overall sample of ninety. Indeed, the sixty applicants who did not come to Yale were just as highly ranked (by us) as those who stayed. Each and every one of them was an exceptional individual with an impressive track record even that early in their medical education. Given this comparability in applicant qualifications, and given that many of the programs they went on to have several commonalities with our program (such as resources, grants and infrastructure), we conclude that the differences in outcomes we found are attributable to the unique characteristics of the AJSP. We go on to outline what we consider to be these ‘active ingredients’.
In developing and refining the AJSP iteratively over its first fifteen years in operation, we have followed a dozen guiding principles we consider integral to its success:
1) Early identity formation. Trainees incorporate direct experience in caring for children and families as they develop as both clinicians and independent physician-scientists;
2) Mentorship and career development. Trainees are assigned a research faculty mentor from their first year onwards. Mentors are accomplished investigators with a sustained record of competitive research funding and active research programs. Mentors have a major responsibility for supervising the trainee, providing assessment and constructive feedback, documenting the trainee’s research progress and performance, and assisting with career development and application for a K award. Participants work closely with their research mentor and residency training director to tailor an appropriate sequence of clinical training and research education and experience;
3) Integrative program structure. The program integrates research with clinical training, and child and adolescent clinical training with adult training by structuring these experiences concurrently and using shared group learning and faculty supervision to foster integration. Unlike traditional training models in psychiatry, both research and child psychiatry training begin early and continue throughout the residency. AJSP trainees are also encouraged to (but not required to have) clinical mentors that guide them to develop particular clinical skills of interest across the training period;
4) Optimal focus on child psychiatry. Wherever possible, child psychiatry rotations are substituted for ones in adult psychiatry, if/as permitted by ACGME and ABPN requirements for both adult and child and adolescent psychiatry. For example, pediatric medicine is scheduled as the required primary care medicine rotation;
5) Foundation of core clinical training. The AJSP provides a full range of inpatient and outpatient experiences that support the acquisition of fundamental clinical skills in adult and child psychiatry. Residents achieve competencies in all six areas identified by the ACGME: medical knowledge, patient care, practice-based learning, interpersonal and communication skills, systems-based practice, and professionalism. They are clinically evaluated using the Milestones of the ABPN. Their strong clinical foundation serves as the basis for evidence-based clinical practice and the development of advanced research skills toward a career as independent investigators;
6) Evidence-based perspective. The principles and practice of Evidence-Based Medicine (EBM) anchor the curriculum and training experiences in both adult and child psychiatry. Regularly scheduled EBM seminars build skills in evidence-based clinical practice;
7) Early research immersion. Intensive immersion in clinical psychiatry research starting during the second year fosters early professional identity development as an investigator and is expected to reduce attrition from long-term commitment to research careers;
8) Formal research training. Optimally, training in the AJSP includes coursework leading to a Ph.D. or master’s degree, if such a degree was not already acquired and/or as determined by a trainee’s learning needs assessment. Concurrent formal research training that is separately supported is available to AJSP trainees through the Investigative Medicine Program or the Yale Department of Epidemiology and Public Health. This formal training can begin as early as the third year;
9) Instruction in responsible conduct of research (RCR). A robust approach to RCR includes formal educational activities supplemented by lectures, workshops and substantial face- to-face discussions. In addition, trainees become familiar with policies and procedures addressing academic misconduct, conflict of interest and conflict of commitment, human subject research protection, and (when relevant), institutional animal care and use;
10) Comprehensive research experience. The AJSP provides a research experience that is comprehensive in terms of time, formal curriculum, mentorship, structured evaluation and feedback. These components are essential for professional growth and development. Over the course of training, AJSP trainees are guided through progressive, supervised research experiences, from critical appraisal of the literature, literature reviews and secondary data analyses, through increasingly complex research projects, independent study design and grant-writing, culminating in the submission of an application for a career development (K-series) award in their final year;
11) Commitment to the enhanced recruitment of under-represented minorities (URMs).The term URM refers to those ethnic or racial groups that are underrepresented in the field of medicine, including African American/African, Hispanic/Latinx, and Native American or Pacific Islander. Eight out of 32 (25%) of AJSP trainees identify as URMs. This fraction is considerably higher than 9%, the AAMC reported national average of URM physicians practicing medicine.29 Multiple reports have also highlighted the severe need for more physician-scientists with URM backgrounds, with only 7% percent of NIH grant awardees being URMs.30 In this context, recruitment of URMs with the intent of improving diversity in the physician and the CAP physician-scientist workforce, has been an important programmatic goal; and
12) Debt repayment. Scheduled research time of at least 80% in the final two years of the AJSP qualifies trainees for the NIH Loan Repayment Program. Thus far, one half of eligible residents enrolled in the AJSP have been able to secure debt loan repayment through this mechanism.
Our study has inherent limitations, beginning with its focus on a single program, which may limit generalizability, particularly to settings with a smaller child psychiatry presence or more limited research infrastructure. In addition, our sample was relatively small, and a fifteen-year window is not sufficient to assess longer-term outcomes, particularly those pertaining to NIH funding. We were not able to collect information on self-reported ethnicity or on current academic or tenure-earning positions for all members of the control group, which limited our ability to determine how our participants and graduates fared comparatively. Finally, we were not able to get individual-level data regarding federal debt relief for control group participants, as the LRP dashboard (dashboard.lrp.nih.gov) provides only state-aggregated information.
In summary, we found that a program specifically designed with the aim of providing specialized training for physician-scientists committed to careers in child and adolescent psychiatry has been able to meet its aims fifteen years since inception. Critical to the program’s fiscal wellbeing are two federal grants that combined provide approximately one quarter of its overall funding (17% and 8% through the R25 and T32 mechanisms, respectively). At a time of uncertainty regarding NIH funding, we consider our findings informative in setting priorities and confirming a solid return on investment. Specifically, the 2.8 million dollars in R25 funding since 2004 has already yielded 4.6 million dollars in new grant funding. Aside from the fiscal bottom line, the AJSP has led to the formation of a unique group of clinician-scientists with remarkable scientific creativity, innovation and output. We look forward to continuing to refine the program and are committed to its longevity. We are hopeful that other programs may consider replicating or expanding the AJSP approach, something that the Universities of Colorado and Vermont have already started doing. There is a pressing public health need for these and other innovative approaches to enhance the training of clinician-scientists devoted to ease the suffering of children and adolescents with psychiatric illnesses.