Survey Analysis to Determine the Impact of Evidence Informed Practice Education upon Faculty Clinical Instruction and East Asian Medicine Student’s Skills, Knowledge, Attitudes and Behaviors

Background Between 2013-2018 Pacic College of Health and Science (formerly Pacic College of Oriental Medicine) trained faculty and developed curriculum in evidence informed practice (EIP), with support from a grant from the National Institutes of Health (NIH). A three-credit (45 hour) Foundations of EIP course, and online EIP learning modules (developed as part of a previous NIH R25 award), were used for faculty and student training. In addition, EIP was incorporated into 73% of the East Asian medicine degree program. Clinical integration of EIP in the College clinic was enhanced by improving access to reference sources, including additional EIP-related questions to the patient intake forms, requiring the use of a patient-centered outcome instrument, and assessing student’s clinical EIP competencies. Methods Master’s degree student’s EIP skills, knowledge, attitudes and behaviors were assessed before and after taking the Foundations of EIP course using a 17-question paper-based survey with an additional open-ended comments section. The survey was administered in 29 courses across all three Pacic College campuses. Clinical faculty EIP instruction was evaluated on the New York City campus using a paper-based survey before and after changes were made to enhance the clinical integration of EIP. A total of 1181 completed EIP-course surveys consisting of 657 pre-EIP course surveys and 524 post-EIP course surveys were analyzed. There was a statistically signicant improvement in student’s EIP skills, knowledge and behaviors after completing the EIP course. Students’ perception of the importance of EIP was high before and after the EIP course. Little change in Faculty’s EIP-related clinical instruction was evident following the EIP-related changes that were made to the Clinic. Students’ self-assessed EIP skills, knowledge and behaviors improved after EIP training. Different approaches may be required to improve implementation of EIP in clinical settings.


Introduction
Evidence-based medicine (EBM) is a foundational principle of healthcare education and practice. De ned as the "conscientious, explicit, and judicious use of current best evidence" along with clinical experience and patient preferences 1 , EBM determines optimal healthcare treatment. Healthcare disciplines based on non-biomedical paradigms and theories, including many of the complementary and integrative health (CIH) therapies, have unique challenges in using EBM. [2][3][4][5] Of primary importance is the scienti c evidence component of EBM. Although there is su cient scienti c evidence to suggest the use of several CIH approaches, [6][7][8] concerns have been raised about external validity and generalizability of the outcomes and conclusions. [9][10][11][12][13][14] It has also led to a degree of skepticism by CIH practitioners. [15][16][17][18] In an effort to increase the research literacy of CIH disciplines and to enhance the use of research in CIH clinical practice, the National Institutes of Health (NIH) has targeted funding toward the development of research training for CIH practitioners.
Between 2013-2018, evidence informed practice (EIP) faculty training and curriculum development was undertaken at the Paci c College of Health and Science (formerly Paci c College of Oriental Medicine, PCOM). This project leveraged a previous NIH NCCIH R25 grant mechanism in which nine CIH institutions were funded to undertake EIP faculty training and curriculum development. 19,20 Paci c College collaborated with faculty from Albert Einstein College of Medicine (Einstein) and Northwestern Health Sciences University (NWHSU) in this endeavor. NWHSU was one of the R25 grant recipients and undertook their EBM project in collaboration with the University of Minnesota (UMN).
As described in a previous publication 21 the approach for this project was to rst train senior faculty, and then through the formation of a faculty led EIP curriculum committee, implement the curriculum revision.
Faculty training was undertaken using online EIP modules 22 developed by NWHSU and UMN, and a threecredit (45 hour) Foundations of EIP course. Both were based on the nine broad EIP competencies that were developed by NWHSU and UMN. 23 Ninety-one percent of the faculty and 97% of the clinical supervisors undertook the EIP training. EIP curriculum development consisted of developing the threecredit (45 hour) Foundations of EIP course, and the introduction of EIP into 73% of the East Asian medicine master's degree program, inclusive of 40 didactic courses and all (15) clinic shifts.
All accredited East Asian medicine colleges are required to have clinics where students ful ll the clinical requirements of their degree programs, which constitute about one-third of the program hours. These clinics serve fee-paying public patients. EIP implementation in the College clinic was facilitated by providing better access to electronic reference sources, adding EIP related questions to patient intake forms, requiring the use of the Measure Yourself Medical Outcome Pro le (MYMOP) 24,25 outcome instrument, and including EIP competencies in students' clinical assessment. Access to reference sources was improved by installing iPads in the Clinic consultation rooms where students and supervisors discuss patient cases and devise treatment strategies. The iPads facilitated access to PubMed and other biomedical internet sites, and were loaded with Chinese medicine text books and Apps.
Surveys were undertaken at the beginning of the project to assess faculty and students' interest and support, 26,27 which showed that faculty and students had a high degree of interest and support for research. However, as students progressed through the four years of the degree program their interest in research declined, which is similar to trends observed at other institutions. 28 A more recent study of Canadian East Asian medicine students in an undergraduate diploma program did not show declines in research interest through the three years of their training. 29 However, this and the US studies, all showed that students had concerns about the appropriateness of scienti c research models for investigating East Asian medicine (model validity). Such concerns stem from the recognition of paradigm and epistemological differences between East Asian medicine and biomedicine. 6,18,29,30,31 As part of this project the perspectives of Paci c College faculty and experienced acupuncturists about research and EBM were investigated and reported in two previous publications -a quantitative survey, 26 and a later qualitative study. 18 Similar to students, faculty expressed concerns about model validity due to paradigm differences and the relevancy of research to clinical practice. They also raised the possibility of cooptation of East Asian medicine by biomedicine. The qualitative study focused on models for combining traditional, complementary and integrative medicine (TCIM) and biomedicine. Study participants, who were experienced acupuncturists (including some Paci c College faculty), showed a de nite preference for a pluralistic model, which is characterized by different healthcare practices harmoniously and respectfully co-existing in a shared healthcare setting that permits each to operate with their own paradigms and methodologies. 18,30 Their discussions also included concerns around model validity, cooptation and power imbalances between TCIM and biomedicine, and the importance of interprofessional education.
Of the nine recipients of the prior NIH NCCIH R25 grants that supported EBM faculty training and curriculum development, only two published their outcomes assessing changes in student's EBM skills, knowledge, attitudes and behaviors in response to EBM training. 32,33 Such outcomes are important for assessing the effectiveness of EBM training programs, their limitations, and for devising approaches for improvement. This study describes the outcomes of the EBM training undertaken at Paci c College. Preand post-course surveys were undertaken with students completing the 3-credit Foundations of EIP course, and clinical faculty were surveyed to assess their EIP-related clinical instruction. The learning, attitude and behavior changes resulting from the EBM training are discussed in relation to the unique challenges of undertaking EBM training with East Asian medicine students and faculty, and in other CIH disciplines.

Methods And Materials
Quantitative EIP Course Survey Students taking the Foundations of EIP course as part of their Master's degree in East Asian medicine were asked to complete a paper-based survey that was developed by researchers at NWHSU as part of their NCCIH R25 grant. The surveys addressed perceived importance of EIP skills, con dence in EIP skills, EIP related behaviors and satisfaction. Questions related to skills were based on the nine EIP competencies that were developed as part of the parent R25 education program. 23 The surveys were administered to students at the beginning of the rst class (pre-survey) and the beginning of the last class (post-survey) of each EIP course.
Pre-and post-surveys had an identical 17 questions (Table 1). Post-surveys also had one additional question (question 18). The rst 12 questions (Table 1) required the survey taker to respond to each question in two parts -A and B. Part A, asked 'How important you think it is for you as a healthcare professional to have the following skills?'. Part B. queried 'How you would rate yourself now in terms of these skills?' Responses to part A assessed EIP attitudes and were a 0-10 Likert scale where 0 equaled 'not important' and 10 equaled 'very important'. Responses to part B assessed EIP skills and knowledge and were a 0-10 Likert scale where 0 equaled 'not competent' and 10 equaled 'very competent'.
Questions 13-16 related to EIP behaviors, and used a 1-5 Likert scale where 1 equaled 'never', 2 equaled 'less than once per month', 3 equaled '1-2 times per month', 4 equaled '3-4 times per month', and 5 equaled 'more than once per week'. Satisfaction was assessed in question 17 by asking 'Overall, how satis ed are you with the quality and content of research-related coursework you have taken at PCOM to date?' and had a 0-10 Likert scale response where 0 equaled 'not satis ed' and 10 equaled 'very satis ed'. The nal question 18 in the post-survey, asked about further research interest -'Would you like to work on a research study as part of your student experience?' and had a yes/no response. The questions were followed by an open-ended comments section.

Clinical Supervisors' Survey
The Clinical Supervisors' Survey was undertaken after the faculty had completed their EIP training. 21 Surveys were completed before (pre) and after (post) changes were made to the clinical policies and procedures to facilitate the use of EIP in the clinic. 21 These changes included the provision of electronic access to reference sources, the addition of EIP related questions to the patient intake forms, the required use of an outcome instrument -the MYMOP, 24

Survey Implementation
Use and implementation of these surveys was approved by the Einstein and Paci c College institutional review boards, and informed consent was obtained from all participants. All methods were carried out in accordance with relevant guidelines and regulations.
The surveys were paper-based. Students completed the pre-and post-EIP course surveys in class. Clinical faculty were asked to complete the clinical supervisors' surveys during a break in the clinical department meetings. For both students and faculty, survey participation was voluntary with no penalty for nonparticipation. Information for matching student pre-and post-course surveys was collected, but no matching information was collected for clinical faculty surveys.
The pre-and post-EIP course surveys were administered in 29 Foundations of EIP courses. All on-ground

Statistical Analyses
Changes in perceived importance of EIP skills, con dence in EIP skills, EIP behaviors, and satisfaction before and after the EIP-course were assessed using linear mixed models with nested random effects for students within each course to account for the correlated nature of the data. Year and location were included in each model to control for differences across campuses and time. Con dence intervals for changes in perceived importance of EIP skills were constructed using bootstrap methods (1000 iterations) due to the skewed nature of their distributions. Importance of EIP skills were considered as a potential confounder for analyses of con dence in EIP skills; however, the relationship between importance of EIP skills and changes in con dence were very weak and were not included as covariates (absolute correlations < 0.12). A sensitivity analysis was performed to assess the impact of limiting the analyses to students with both pre and post-EIP course surveys.
For the clinical supervisors' surveys, we assessed how frequently faculty used EIP in the clinic before and after EIP training. Differences in the distribution of faculty reporting varying levels of EIP use pre-and post-training were assessed using a chi-square test.

Qualitative Analysis of Open-Ended EIP Course Survey Question
Comments that were written in the open-ended comments section of the pre-and post-EIP course surveys were transcribed into a spread sheet document by SD. BA and SD read through the comments and met to discuss emerging themes. SD categorized the comments into 29 categories. BA read through these categories and condensed them into six themes. SD and BA evaluated and discussed these themes, and decided upon the nal six themes presented in this paper.

Quantitative EIP Course Survey
Survey data was derived from students in 29 Foundations of EIP courses -17 on the NY campus, eight on the San Diego campus, and four on the Chicago campus. A total of 1181 completed EIP-course surveys consisting of 657 pre-EIP course surveys and 524 post-EIP course surveys were analyzed. The survey response rate among the various EIP courses was over 90%. An exact calculation of the response rate was not possible due to slight uctuations in student numbers as a result of students dropping and adding the courses. Findings from the sensitivity analyses, which only included students with matched pre and post surveys (n = 233), were very similar to the primary analyses in terms of magnitude of change The pre-and post-EIP course survey responses are shown in Table 1. When asked about the importance of EIP (Table 1A) the scores ranged from 8.2 to 9.1 in the pre-EIP course data, and from 8.2 to 9.2 in the post-EIP course data. Four questions showed a statistically signi cant increase in the pre to post importance rating. Scores for self-reported competence (Table 1B) ranged from 4.0 to 5.7 in the pre-EIP course data, and from 6.9 to 8.5 in the post-EIP course data. All 12 questions showed a statistically signi cant increase in competence.   When asked about EIP behaviors (Table 1C) the scores ranged from 1.9 to 2.8 in the pre-course data, and 3.1 to 3.8 in the post-course data. All four questions showed a statistically signi cant increase in EIP related behaviors. Scores for overall satisfaction with research-related course work at the College (Table 1C) increased from 5.6 to 8.3 in the pre to post assessment, which was a statistically signi cant increase.
Question 18 was only on the post-EIP course survey and asked about further research interest with a yes/no response. Of the 366 respondents 117 (32%) answered yes, 41 (11%) answered no, and 208 (57%) did not answer this question.  post-course surveys that were included in this analysis, the response rate for this open-ended question was 14% for the pre-course surveys and 23% for the post-course surveys. Comments often contained content that was categorized into more than one of the six themes. Therefore, the number of comments listed in Table 2 for the pre-and post-EIP course surveys exceeds the 213 individual comments.
*Comments from individual respondents were frequently categorized into more than one theme. Therefore, the number of comments exceeds the number (n) of respondents.
Total comment percentages were calculated by dividing the total number of comments for each theme by the total number of comments across all themes (298) The majority (31%) of the comments were expressing satisfaction. Most of these were recorded postcourse and focused on the class, instructor, education and learning experience. An example of a postcourse comment was "Awesome class, very fundamental, & I think everyone should takes this course. Other classes should follow this example instructor." The next most frequent theme (22%) were comments expressing a positive view of research. These included comments indicating that the course changed research perspectives, showed the value of research in legitimizing the profession, generated an interest in research, and imparted useful skills. Most of these comments appeared in the post-course surveys. Examples of comments for this theme were -"Excellent course, very informative; I've been able to apply methods learned in this class to other classes, as well as clinic", and "Good class, I learned how to critically assess research in a presentable and systematic way, tailored to my speci c needs (as a future practitioner)." Sixteen percent of the comments were expressing dissatisfaction and were mainly in the post-course surveys. Dissatisfaction was with the length of the course, its level of challenge (both too much and too little), placement sequence in the program curriculum, use of online modules for part of the course, and the need to be more clinically experienced. Examples of comments were -"FEIP is a very important subject for health care professionals, however the actual class was intensely overcomplicated resulting in most of the class not caring about the subject matter", and "This is very important class, but I think it was given on elementary level, not up to par with adults of masters level professionalism".
In the pre-course surveys we had a predominance of comments comparing this course to others in the degree program. These comments mentioned that other courses the students had taken did not integrate research and presented unscienti c perspectives, and a preference for East Asian medicine courses. Examples of this theme were -"I cannot name a class that places research-related coursework at more than 5% of total grade, and I do not value it personally at all", and "I understand the validity of published work in the eld but hope to see more of the classics involved & less of a westernized approach" There were two minor themes. Ten percent of comments referenced prior learning and were mainly in the pre-course comments. These commented on being a transfer student, and having little prior research and clinical experience. Seven percent of the comments expressed a negative view of research and were in both the pre-and post-course surveys.

Clinical supervisors survey
Data from the clinical supervisors' survey is shown in Table 3. Eighteen pre-surveys and 28 post-surveys were completed representing 49% and 76% of the 37 supervisors, respectively. The range of responses to the nine identical questions in the pre/post surveys are shown in Table 3A. The only question that showed a statistically signi cant change from the pre to post survey was in relation to asking students to cite articles, which increased in the post survey. Activities that appeared to be taking place the most frequently were comparing and contrasting different approaches, identifying sources of information when teaching, and providing references to students. Responses to the three additional questions on the post-survey are shown in Table 3B.  sources, and the requirement for more EIP-related documentation in clinic intake forms and student assessment.
The pre-and post-mean scores (Table 1) for the student's assessment of the importance of EIP skills (re ective of their EIP attitudes), and their self-reported EIP competence (EIP skills and knowledge) showed little variation between the questions. Pre-scores for competence were slightly higher for questions relating to nding and incorporating research into their assignments and coursework possibly because they had been using these skills in other courses. Mean scores for the question asking about nding ways to actively participate in research, were slightly lower.
Responses to questions assessing student's EIP behaviors (Table 1C) showed that before the Foundations of EIP course they were discussing research, accessing summary sources, and seeking research to answer clinical questions on average once per month, and applying the EIP model less frequently. After the Foundations of EIP course there was an increase to at least once every other week in the frequency of accessing summary sources and applying the EIP model. There was also an increase in the frequency of discussing research and seeking research to answer clinical questions. At the end of the course students appeared to be discussing research with others every week.
The trend of recognizing the importance of EIP and research has been shown in our and several other prior studies with both East Asian medicine students and faculty. 18,26−29 In these studies, the importance of research was mainly with respect to those outside the profession -the public, referring providers, and insurance companies. Two of the three studies, at three different East Asian medicine colleges, also showed that student's interest in research, and its perceived value to clinical practice, declined as they progressed through their training. 26,28 This same trend has also been reported with chiropractic students. 32 This trend of valuing the importance of research for those outside the profession may be the reason why the students in this survey scored the importance of research skills and knowledge above 8 on the 0-10 Likert scale both before and after taking the EIP course. Taking the course did not diminish their value of research importance. In fact, comments in the open-ended survey question expressing a positive view of research were much more frequent in the post-course surveys, and only 7% of the pre-and post-course comments expressed a negative view of research. This same trend was seen in the study of Canadian East Asian medicine students where their perception of the importance of research did not diminish after taking a 30-hour research literacy course. 29 This contrasts with the prior studies, 26,28 which showed that student's perception of the importance of research to their clinical practice declined as they progressed through their training. An important difference between these studies is that in this study the perceived importance of research skills and knowledge was assessed before and after a single course over the period of a few months, whereas the previous studies 26, 28 assessed changes throughout a degree program over three to four years. In the latter, students likely had more exposure to the clinical practice of East Asian medicine, and therefore greater opportunity and experience to determine the differences between treatments administered in clinical trials and those performed in real-world clinical settings.
Another possible explanation for the difference between this and the Canadian study, compared with the previous studies, is the way in which research methodology has changed over the past decade. Clinical research in East Asian medicine started with e cacy trials and use of the randomized placebo-controlled trial. Problems associated with the validity of this approach have been widely discussed in the literature for both TCIM therapies 29,34−36 and biomedicine. 37 For East Asian medicine many of the fundamental ways in which the medicine is practiced are not included in these randomized controlled trials, for example, differential East Asian medicine diagnosis, individualized treatments, and the inclusion of a range of different modalities in addition to acupuncture. The use of non-inert placebo controls often results in there being no statistically signi cant difference between verum and placebo acupuncture. 38 The predominance of these types of trials has led to suspicion and disregard of research by East Asian medicine practitioners, [15][16][17][18]  shifts they provided references for treatment to their students, asked students to compare and contrast different approaches to treatment, and identify sources of information. The only activity that showed a signi cant increase after EIP related changes were made in the College clinic was requiring students to cite articles, however the frequency of this activity remained low. The supervisors reported using the electronic resources themselves and asking students to use them in about half of their clinic shifts.
MYMOP use did not occur frequently. This was expected because MYMOP was only required to be used on patients coming to the College clinic for the rst time, which on most shifts would have occurred far less frequently than patients returning to the clinic for ongoing treatments. Therefore, we concluded that MYMOP had been adequately adopted.
The trends seen in the outcomes of the Clinical Supervisor's survey suggest that using scienti c research sources to inform clinical practice was less prevalent than comparing and contrasting different treatment approaches and identifying information sources, the latter most likely originating from East Asian medicine literature sources. These outcomes may have been due to insu cient EIP training. However, other explanations are also worthy of consideration. A bias towards greater reliance on East Asian medicine sources to inform treatment would be expected because students and faculty are trained in this discipline. During the thirteen one-hour meetings of the EIP Curriculum Committee, which occurred over a 2.5-year period, much discussion occurred around the meaning of the word evidence. 21 The committee felt strongly that evidence needed to be interpreted broadly, and that East Asian medicine textbooks were an important resource for justifying treatment approaches. Preference for these sources is signi cantly magni ed by the challenges associated with applying the scienti c research due to the model validity issues discussed above. However, it is possible that even with research that very accurately resembles real-world practice there will always be a preference for East Asian literature sources. Previous studies of ours 18,26 and others 39 42 Behavioral change models suggest that motivation to adopt EBM is needed along with EBM education and the provision of resources, 43 and therefore it is likely that different approaches will be needed to engender greater EIP clinical activity. Additional formative assessment is needed to determine the impact of the EIP training upon faculty and student's adoption of EIP in the Clinic, and to identify barriers associated with skills, knowledge, availability of resources and time, attitudes and motivation. Such studies may also shed light on how to develop more effective strategies to engender greater use of EIP in clinical activities, some of which may be those developed in the previous NIH R25 grants. 19 This study has limitations. Being an observational study any improvements in EIP skills, knowledge, attitudes and behaviors cannot be conclusively attributed to the EIP training. Linkage of pre-and post-EIP course surveys was done using three coding questions ( rst 2 letters of Mother's rst name, day of birth month, and last letter of rst name). Due to the inconsistency of answers we were only able to match 233 of the 1181 completed surveys. This limitation did not have a large impact on our results as sensitivity analyses limited to matched participants had very similar ndings to our primary analyses using all available data. The response rate to the open-ended question was 14% for the pre-course surveys and 23% for the post-course surveys, and opinions expressed may not accurately represent all students' perspectives. The surveys used in this study were not psychometrically tested. The Clinical Supervisors survey was administered by the Academic Dean and thus faculty may have felt obligated to respond in a favorable manner. The response rate for this survey was 49% for the pre-and 76% for the post-survey, and may not accurately represent all faculty's responses. Given that research training is not an accreditation requirement for East Asian medicine programs at the master's degree level, the outcomes at Paci c College may not be typical for all East Asian medicine colleges in the US. All self-reporting has inherent inaccuracy, and previous studies have shown that EBM surveys are sometimes associated with inaccurately elevated self-reported EBM literacy. 44

Conclusions
Our study demonstrated that East Asian medicine students recognize the importance of EIP, and that a three-credit EIP course may be effective at improving EIP skills, knowledge and behaviors. Training clinical supervisors and providing greater access to evidence sources in the College clinic did not appear to increase EIP instructional activity. Therefore, it is likely that different approaches are required to improve implementation of EIP in clinical settings. Availability of data and materials:

Abbreviations
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
K07AT007186 and R25AT003582. This publication was also supported in part by the Clinical and Translational Science Award Grant 1 UL1 TR001073-01, 1 TL1 TR001072-01, and 1 KL2 TR001071-01 from the National Center for Advancing Translational Sciences, United States, a component of the NIH.
The content is solely the responsibility of the authors and does not necessarily represent the o cial views of the NIH.
Authors contributions: BA, RE, BK and PM conceived the research project and design of the study, RE supplied the student survey instrument, which had been used by her and colleagues in previous similar research studies, BA and SD undertook the study, SD undertook the data collation and preliminary analysis, BL undertook the statistical analysis of the data, all authors were involved in interpreting the data and preparing the manuscript.