Our study showed that Pacific College students had a high degree of recognition of the importance of EIP before taking the Foundations of EIP course, and that this remained high after completing the course with little to no change in perceived importance. Responses to the open-ended survey question also suggest that the Foundations of EIP course may have engendered a positive view of research among the students. Their self-rated EIP competence improved significantly after completing the course as did all EIP behaviors. A high level of satisfaction with the course was reflected by their response to the satisfaction survey question, and in the open-ended responses. The practice of EIP by the clinical supervisors was moderate, and was little changed after the provision of improved access to reference sources, and the requirement for more EIP-related documentation in clinic intake forms and student assessment.
The pre- and post-mean scores (Table 1) for the student’s assessment of the importance of EIP skills (reflective of their EIP attitudes), and their self-reported EIP competence (EIP skills and knowledge) showed little variation between the questions. Pre-scores for competence were slightly higher for questions relating to finding and incorporating research into their assignments and coursework possibly because they had been using these skills in other courses. Mean scores for the question asking about finding ways to actively participate in research, were slightly lower.
Responses to questions assessing student’s EIP behaviors (Table 1C) showed that before the Foundations of EIP course they were discussing research, accessing summary sources, and seeking research to answer clinical questions on average once per month, and applying the EIP model less frequently. After the Foundations of EIP course there was an increase to at least once every other week in the frequency of accessing summary sources and applying the EIP model. There was also an increase in the frequency of discussing research and seeking research to answer clinical questions. At the end of the course students appeared to be discussing research with others every week.
The trend of recognizing the importance of EIP and research has been shown in our and several other prior studies with both East Asian medicine students and faculty.18,26−29 In these studies, the importance of research was mainly with respect to those outside the profession – the public, referring providers, and insurance companies. Two of the three studies, at three different East Asian medicine colleges, also showed that student’s interest in research, and its perceived value to clinical practice, declined as they progressed through their training.26,28 This same trend has also been reported with chiropractic students.32
This trend of valuing the importance of research for those outside the profession may be the reason why the students in this survey scored the importance of research skills and knowledge above 8 on the 0–10 Likert scale both before and after taking the EIP course. Taking the course did not diminish their value of research importance. In fact, comments in the open-ended survey question expressing a positive view of research were much more frequent in the post-course surveys, and only 7% of the pre- and post-course comments expressed a negative view of research. This same trend was seen in the study of Canadian East Asian medicine students where their perception of the importance of research did not diminish after taking a 30-hour research literacy course.29 This contrasts with the prior studies,26,28 which showed that student’s perception of the importance of research to their clinical practice declined as they progressed through their training. An important difference between these studies is that in this study the perceived importance of research skills and knowledge was assessed before and after a single course over the period of a few months, whereas the previous studies26,28 assessed changes throughout a degree program over three to four years. In the latter, students likely had more exposure to the clinical practice of East Asian medicine, and therefore greater opportunity and experience to determine the differences between treatments administered in clinical trials and those performed in real-world clinical settings.
Another possible explanation for the difference between this and the Canadian study, compared with the previous studies, is the way in which research methodology has changed over the past decade. Clinical research in East Asian medicine started with efficacy trials and use of the randomized placebo-controlled trial. Problems associated with the validity of this approach have been widely discussed in the literature for both TCIM therapies29,34−36 and biomedicine.37 For East Asian medicine many of the fundamental ways in which the medicine is practiced are not included in these randomized controlled trials, for example, differential East Asian medicine diagnosis, individualized treatments, and the inclusion of a range of different modalities in addition to acupuncture. The use of non-inert placebo controls often results in there being no statistically significant difference between verum and placebo acupuncture.38 The predominance of these types of trials has led to suspicion and disregard of research by East Asian medicine practitioners,15–18 and this perspective may be the predominant perspective seen in the earlier studies where students and faculty questioned the relevance of the research to clinical practice.
Due to the recognition of the limitations of the randomized placebo-controlled trial to accurately inform effectiveness in the real-world, different research models have been developed. These include pragmatic clinical trial models,10 comparative effectiveness methodology, whole systems research,29 and mixed methods. In addition, outcome measures beyond those focused on the condition under investigation, such as quality of life measures, are now routinely used. These allow the broader clinical benefit to also be assessed. Trials using these approaches are just beginning to predominate the recent research literature. The EIP course developed in this project,21 and that used in the Canadian study,29 included content on these new research models, along with discussions about the weaknesses of the randomized placebo-controlled trial, and its deleterious impact on the evaluation of the effectiveness of East Asian medicine. This, combined with the presence of such trials showing up in student’s literature searches, may also explain why we did not see declines in student’s perception of research importance before and after taking the research courses.
The Clinical Supervisor’s survey was undertaken after the faculty had received EIP training.21 This training utilized the online EIP modules22 developed by NWHSU and UMN through their R25 grant, and in-service trainings at department and general faculty meetings. Although 97% of the clinical supervisors undertook the EIP training, 65% only completed 4 of the 10 modules, which was the minimum EIP training requirement. The survey indicated that clinical supervisors seldom required students to cite articles, discuss how to conduct a literature review, compare and contrast scientific articles, discuss ideas about research, and discuss the strengths and weaknesses of research. However, in about half of their clinic shifts they provided references for treatment to their students, asked students to compare and contrast different approaches to treatment, and identify sources of information. The only activity that showed a significant increase after EIP related changes were made in the College clinic was requiring students to cite articles, however the frequency of this activity remained low. The supervisors reported using the electronic resources themselves and asking students to use them in about half of their clinic shifts. MYMOP use did not occur frequently. This was expected because MYMOP was only required to be used on patients coming to the College clinic for the first time, which on most shifts would have occurred far less frequently than patients returning to the clinic for ongoing treatments. Therefore, we concluded that MYMOP had been adequately adopted.
The trends seen in the outcomes of the Clinical Supervisor’s survey suggest that using scientific research sources to inform clinical practice was less prevalent than comparing and contrasting different treatment approaches and identifying information sources, the latter most likely originating from East Asian medicine literature sources. These outcomes may have been due to insufficient EIP training. However, other explanations are also worthy of consideration. A bias towards greater reliance on East Asian medicine sources to inform treatment would be expected because students and faculty are trained in this discipline. During the thirteen one-hour meetings of the EIP Curriculum Committee, which occurred over a 2.5-year period, much discussion occurred around the meaning of the word evidence.21 The committee felt strongly that evidence needed to be interpreted broadly, and that East Asian medicine textbooks were an important resource for justifying treatment approaches. Preference for these sources is significantly magnified by the challenges associated with applying the scientific research due to the model validity issues discussed above. However, it is possible that even with research that very accurately resembles real-world practice there will always be a preference for East Asian literature sources. Previous studies of ours18,26 and others39 have shown that faculty and practitioners are concerned about power imbalances between TCIM and biomedicine, and the possibility of cooptation. The validity and historical underpinnings of such issues have been widely discussed in the literature.2,3 These are additional and powerful attitudes that could sustain preference for East Asian medicinal sources.
The significant and consistent increase in student’s self-assessed competence suggested that their EIP skills and knowledge improved as a result of taking the Foundations of EIP course. The development of this course was based on a very similar course that had previously been developed at NWHSU and was shared with Pacific College as part of the collaboration for the NIH grant. Similar improvements in NWHSU’s students EIP skills and knowledge as a result of completing the Foundations of EIP course have been reported.33 This strongly suggests that this course was useful for imparting EIP knowledge and skills, and that this was sustained across different institutions and instructors. Similar increases in student’s EIP knowledge following the completion of EBM courses have been reported by the other NIH R25 grant recipients19.19,32
At both NWHSU and Pacific College increases in EIP behaviors were also seen after completing this course. However, these studies don’t indicate the extent to which this translated into an increase in EIP-related activities in the College clinics as a result of this training. Previous studies have shown that successful EBM training does not usually change EBM attitudes and behaviors for both TCIM32,40,41 and biomedical practitioners.42 Behavioral change models suggest that motivation to adopt EBM is needed along with EBM education and the provision of resources,43 and therefore it is likely that different approaches will be needed to engender greater EIP clinical activity. Additional formative assessment is needed to determine the impact of the EIP training upon faculty and student’s adoption of EIP in the Clinic, and to identify barriers associated with skills, knowledge, availability of resources and time, attitudes and motivation. Such studies may also shed light on how to develop more effective strategies to engender greater use of EIP in clinical activities, some of which may be those developed in the previous NIH R25 grants.19
This study has limitations. Being an observational study any improvements in EIP skills, knowledge, attitudes and behaviors cannot be conclusively attributed to the EIP training. Linkage of pre-and post-EIP course surveys was done using three coding questions (first 2 letters of Mother’s first name, day of birth month, and last letter of first name). Due to the inconsistency of answers we were only able to match 233 of the 1181 completed surveys. This limitation did not have a large impact on our results as sensitivity analyses limited to matched participants had very similar findings to our primary analyses using all available data. The response rate to the open-ended question was 14% for the pre-course surveys and 23% for the post-course surveys, and opinions expressed may not accurately represent all students’ perspectives. The surveys used in this study were not psychometrically tested. The Clinical Supervisors survey was administered by the Academic Dean and thus faculty may have felt obligated to respond in a favorable manner. The response rate for this survey was 49% for the pre- and 76% for the post-survey, and may not accurately represent all faculty’s responses. Given that research training is not an accreditation requirement for East Asian medicine programs at the master’s degree level, the outcomes at Pacific College may not be typical for all East Asian medicine colleges in the US. All self-reporting has inherent inaccuracy, and previous studies have shown that EBM surveys are sometimes associated with inaccurately elevated self-reported EBM literacy.44