DOCENT: A Direct Observation Clinical Experience with feedback iN real-Time (DOCENT)

Observation of student skills is essential for accurate assessment of entrustment and competence. Time spent directly observing students in patient care must balance with the need to serve adequate numbers of patients and, in some instances, revenue generation. Clinical education may also be hampered by a negative learning climate. The authors created a Direct Observation Clinical Experience with feedback iN real Time (DOCENT) clinic with patients from the Emergency Department (ED) to provide care for low-acuity patients while observing student care to determine the best location for the clinic and interprofessional education opportunities.


Introduction
Competency-based education is a mainstay of medical education (1). The American Association of Medical Colleges (AAMC) recognized critical areas of competence when establishing thirteen core entrustable professional activities (EPAs). Direct observation is essential to determining entrustment; in 2019, the AAMC inquired about the frequency of direct observation for each EPA on the graduation questionnaire, a survey completed by every graduating medical student. Direct observation of students in patient work ow was potentially impacted changes from the Center for Medicare and Medicaid services (CMS) impacting criteria for billable services involving students. While CMS now allows for student documentation to be used by billing providers for fee-based services, it also mandates direct observation of students in patient care encounters.
Unfortunately, academic faculty report barriers to teaching and direct observation, including demands of patient care, competing academic priorities, burnout, and lack of recognition and support (2). Provision of speci c and actionable feedback, which is necessary to impact learners (3); requires educator training and investment which is often not possible given time constraints in the clinical environment (4).
In an effort to solve these problems, some suggest assigning activities to students that "add value" to the team, thereby reducing workload to the team overall (5), while others suggest hiring coaches to improve technical and nontechnical skills (6). Teaching at the bedside is thought to be a particularly effective way to provide feedback to learners (7). Some have created student-run clinics to meet these educational needs; while these have shown educational outcomes (8), the clinics are infrequent, voluntary for students, and some have questioned ethics of clinic suggesting that the clinics take advantage of vulnerable populations (9).
Our goal was to create a mandatory direct observation clinical experience where faculty were compensated directly for teaching; provision of feedback was an expectation for faculty and students, and patient care did not suffer. Speci cally, the clinic needed to provide care that did not differ signi cantly from care provided to anyone else who accessed the health system from through the emergency department. Secondary goals included providing the opportunity for peer-to-peer teaching and modeling interprofessional interactions. programs. The group sought to provide care to patients already seeking medical care within our health system that would bene t from a teaching team: patients with low estimated severity of illness (ESI) in the Emergency Department (ED). These patients often had long wait times and sometimes left without being seen. Evening hours were selected as the appropriate time since this is a busier time in the ED. The timing of DOCENT for the rst year (December 2015-December 2016) was on weeknights (Monday-Friday) from 5:00pm-9:00pm. This timing allowed preclinical and clinical student to participate without disrupting other scheduled educational activities and faculty involvement without interfering with other clinical and/or teaching responsibilities. Faculty participation was based on interest in medical education and reputation for excellence in teaching.

Physical Space/Logistics
The physical location for DOCENT (December 2015-December 2016) was an underutilized outpatient clinic space already staffed with nurses (an infusion clinic that operated until 9:00pm nightly). Patients for DOCENT were triaged in the ED, assigned an estimated severity of illness (ESI score), and a medical screening exam was performed by Emergency Medicine physicians. Inclusion and exclusion criteria for patients included: non-pregnant patients > age 18, an ESI score > 3, and not expected to require imaging. Eligible patients were consented to participate in DOCENT or could choose to remain in the ED. Once consented, patients were walked to the clinic space by DOCENT students and faculty. Clinic faculty logged all patients' chief complaints. Patient presence in the clinic was tracked via the electronic health record (EHR) when a patient was moved to a bed a liated with DOCENT.
From January 2017 to June 2017, the clinic was moved to a physical location closer to the ED in a space that was otherwise only lled during daytime hours. The close proximity to the ED expanded the number of eligible patients by making it easier to see patients with higher assigned ESIs and those who might need radiologic imaging. Faculty facilitating DOCENT selected the patients, completed medical screening exams, and consented the patients. Nurses from the ED provided nursing coverage. The nights of operation changed from Mondays to Thursdays in order to free up funds to support nursing coverage. Further, many patients arriving at the ED on Friday night were sent to the ED by outlying clinics for admission, consultation, or additional testing; completing encounters during DOCENT hours was di cult. Internal Medicine subspecialty. The number of shifts were equally divided between EM and non-EM faculty. The SOM also paid an ED nurse (commensurate to their ED hourly rate plus overtime) to staff the clinic and provide interprofessional perspectives on care. Like faculty, nurses worked in the clinic in addition to their scheduled ED shifts. Two faculty physicians served as "Co-course Directors" for DOCENT. They received academic time adjustment and partial time from a coordinator. Together, the course directors and coordinator scheduled students and faculty, developed orientation materials, and supported faculty development. Revenue generated from patient encounters was received by the health system and used to support clinic infrastructure/emergency department function.

Methods/Outcomes
DOCENT patients were tracked manually via a clinic log. After the 300th patient, the types of complaints were reviewed and categorized retrospectively by leadership. Thereafter, the category of patient complaint was logged prospectively. Patient speci c data were tracked by the health system for quality improvement purposes and subsequently reviewed for this study, including ESI, length of stay, use of diagnostic imaging, and payer. Clinic patients were identi ed after being "moved" electronically in the EHR to the DOCENT "space". ED patients were selected as a comparison group if they had a similar severity of illness and presented during weekdays between 3:00pm and 7:00pm, the equivalent period of time during which DOCENT patients were triaged.
During the rst year, patients seen as part of DOCENT were contacted by telephone to assess satisfaction with the care provided. Subsets of the clinic population and comparison patients from the ED were asked to complete standard surveys sent by the health system after care was completed. Four Likert-type questions were asked with possible responses of 'very poor', 'poor', 'fair', 'good', and 'very good': 1) How well was your team's concern for your comfort? 2) How well did the team take time to listen to you? 3) How well were you kept informed of delays? and 4) How likely would you be to recommend this experience?
After clinic operations and logistics were nalized, efforts were made to optimize and evaluate student experiences. Learners participating in clinic were asked to complete evaluations. Medical students received written and verbal reminders regarding the completion of the evaluation. Students were asked if they were directly observed obtaining a history, performing a physical exam, and creating differential diagnosis and/or management plans (yes/no). Since multiple students worked in the clinic together, it was possible for one student to report not being directly observed for one aspect of the encounter, because the other student was responsible for that part of the counter. Students were also asked if they received reinforcing or constructive feedback using a Likert item (no, somewhat, yes). Students reported if they worked with a student from another profession, and clinical students were asked if they had the opportunity to supervise preclinical students.
This study was reviewed and received exempt status from the Duke Health Institutional Review Board.

Data Analysis
Patient care experiences in DOCENT vs. the ED were evaluated using a Kruskal-Wallis test to compare length of stay, Chi-square tests to compare distributions for illness severity and number of patients receiving radiologic imaging and Fisher's exact tests for patient satisfaction. Students' experience of DOCENT was summarized with survey questions and compared between preclinical (1st year) and clinical (3rd year) students. A two-sided signi cance level of 0.05 was used for all statistical tests without multiple comparison adjustment. Statistical analyses were performed using SAS version 9.4 (SAS Institute, Inc., Cary, NC, USA) and R version 3.4.0 (R Core Team, Vienna, Austria) for Windows.

Patient Care Experiences/Outcomes during the Optimization Period
In the rst 18 months of the clinic, a total of 532 patients were seen as part of the direct observation experience for medical students and a total of 11,127 ED patients were selected as a comparison group. These DOCENT patients had a shorter or analogous length of stay when compared to ED patients (mean 4.5 vs. 6.4 hours, p < 0.0001) with similar acuity and time of day at presentation to the ED.
DOCENT patients reported higher satisfaction than those patients who remained in the ED. Comparisons were initially made with completed paper surveys of 14 DOCENT patients with 336 ED patients who completed the same survey. The number of patient satisfaction surveys from DOCENT was small because DOCENT patients represent a very small percentage of all patients seen in the ED. Therefore, we also compared DOCENT patient responses from a telephone survey with the responses from ED patients. Comparisons were made between 337 ED patients who completed paper patient satisfaction surveys and 126 DOCENT clinic patients who completed follow-up phone calls with the same questions (see Fig. 1).
This was the same trend seen when comparing 14 DOCENT patients with the 336 ED patients, however this N was too small to perform statistical analysis.

Location and Outcomes
When the DOCENT was moved from a traditional outpatient-type clinic space to a location closer to the ED, higher acuity patients were seen (see Fig. 2). Although acuity increased, patient length of stay was similar between the clinic spaces further away from and closer to the ED (M = 4.6 hours, SD = 2.

Types of Patients Seen By Students
After the nal location was established (October 2016-June 2018), the chief complaints of 423 patients were evaluated in the direct observation clinical experience (Fig. 3). The most common complaints were musculoskeletal (MSK). The number of MSK symptoms evaluated by health care teams may be in uenced by the optional and voluntary participation of students and faculty from the Doctor of Physical Therapy program who work alongside medical students as often as once per week.

Student Feedback about Experience
From October 2017 to December June 2018, 421 medical students completed evaluations of DOCENT, including 238 preclinical (1st year) and 183 clinical (3rd and 4th year) students. A higher proportion of preclinical students reported that they were observed obtaining history than clinical students (98.3% vs. 88.0%, Chi-square test p < 0.0001). Alternatively, when compared to preclinical students, a higher proportion of clinical students reported that they were observed generating a prioritized differential diagnosis (94% vs. 76.5%, Chi-square test p < 0.0001) and developing a management plan (92.3% vs. 60.5%, Chi-square test p < 0.0001) (see Table 1). Over 85% of both preclinical and clinical students (> 85%) had the opportunity to be observed performing physical exams (Table 1).
Both preclinical and clinical students reported that faculty provided reinforcing and constructive feedback and created a positive learning environment (see Fig. 4). More students reported receiving reinforcing feedback than constructive feedback (85.7% vs. 78.1%, p = 0.0162).

Student-Student Interactions in the Clinic
Ninety-four percent of clinical medical students had the opportunity to participate in peer-to-peer teaching with preclinical students. Sixty-percent of medical students worked in the clinic with a student from another profession and 53% with faculty from another profession (n = 445).

Study Limitations
There are limitations to this study. Length of stay data were likely impacted by selection bias as faculty selected patients within their scope of practice and whose encounters were estimated to be completed during a clinic session. This may have resulted in seeing patients within the ESI level who were less complicated, thus affecting length of stay. Patient satisfaction data were impacted by response rates, including only those patients who took phone calls or completed written satisfaction surveys.
Additionally, the difference in survey modality (phone call vs. written) could skew the response data.
Student evaluations of the clinic may have been impacted by the quality of teachers who were asked to participate in the clinic. Only faculty who were known to be exceptional teachers were invited to participate. This select pool was narrowed further as only those faculty members who were interested in the mission of the clinic agreed to work in the clinic.
One major factor that likely affected the data collected on numerous levels was the type of individual faculty who participated in the clinic. For example, patient selection may vary based on the background of the faculty and thus, their experience and comfort with the various complaints that could be encountered in the ED waiting room. Additionally, each faculty provider has their own tolerance for risk and thus, one provider may order more radiologic or laboratory tests for a complaint than others would for the same complaint. These same factors could in uence student engagement and perception of the care provided. Finally, it is possible that the patients' satisfaction with the experience were also impacted based on the provider's background and comfort with treatment modalities (e.g., use of speci c stretches and exercises for MSK pain as opposed to relying on medications only).

Discussion
We have successfully created a direct observation clinical experience for medical students that did not adversely impact health system outcomes and which positively impacted patient satisfaction with zero additional cost to the health system. Students had high rates of satisfaction, reported direct observation of clinical skills commensurate with level of experience and received timely and speci c feedback. This clinic structure represents a win-win for the health system and the school of medicine. The health system offsets the cost of patient care, since patients were already being seen in the emergency department. The SOM offset the costs of adding additional personnel to provide care to ED patients. Faculty were easy to recruit since the experience was not dictated by throughput of patients (generation of clinical RVUs). Patient care was not different from care provided to patients who remain in the emergency department (the same resources are available for all patients), reducing the risk of taking advantage of vulnerable populations.
Patients also bene t from the DOCENT clinic. While LOS did not increase (and for in some cases decreased), patients seen in DOCENT do spend a greater percentage of the time in direct contact with the health care team compare to patients who remained in the ED. Since the clinic space does not have separate "workrooms", students obtain history and perform physical exams with faculty in the room. The team then does not leave the room to discuss patients' history, their complaints, ndings from the physical exam, the clinical reasoning for differential diagnoses, and potential management plans. Rather, students think aloud in the presence of the patients, potentially improving communication, including shared decision-making. This may contribute to higher patient satisfaction.
When considering the ideal location for DOCENT, the ED is ideal for both students, patients and the health system. There is usually an abundance of patients in the ED. These patients have undifferentiated complaints, allowing students to walk through clinical reasoning and develop differential diagnoses, additional testing, and management plans. Patients with lower acuity ED complaints may have been utilizing the ED because of complex social needs (outpatient option availability, mental illness, insurance status, etc.), which allows for discussion of social determinants of health and health disparities that may be ignored in a busy clinical setting. With regard to physical space, the outpatient-type clinical space, with already available nighttime nursing, allowed teams to see patients who needed laboratory testing or medication administration. However, a medical screening exam needed to be performed prior to walking to that space to complete the encounter and prevented those who may have needed radiologic studies, treatment modalities, or multiple tests from being considered. This limited the number and types of patients seen. In contrast, the location adjacent to the ED allowed faculty the opportunity to see patients with higher acuity and more complexity, including those undergoing advanced imaging (CT or ultrasound) who may have needed multiple tests or more than just oral medications.
The DOCENT experience demonstrates that direct observation of core entrustable professional activities is possible when clinical experiences are focused on education and not pressured by clinical revenue. By shifting the stated purpose of the clinic from clinical e ciency and nancial solvency to focusing primarily on education and teaching, faculty can be held to a different standard in this unique clinical experience that includes being held responsible for participating in faculty development, providing real-time feedback to students and completing on-time evaluations of students in clinic. While most clinical experiences have some form of observation and feedback, our student evaluations show that students recognize these are occurring and that observation and real-time feedback are fundamental and valued in the experience. These evaluations also show that faculty are able to adapt their expectations for students in different years of medical school. As expected, preclinical students are observed more often taking a history, while clinical students are observed developing differential diagnosis and management plans. Most learners were also observed performing physical exams.
One area for potential improvement is helping faculty to focus on providing more constructive feedback since this type of feedback was given less frequently than reinforcing feedback. The DOCENT clinic also provided an opportunity for clinical students to teach their peers. Not only is peer-to-peer teaching effective (10), but peer-assisted learning in a student-run free clinic project increased clinical competence (11). As interns, students are expected to teach their medical student near-peers (12). Giving students the opportunity to teach their peers under direct faculty supervision in DOCENT is a unique opportunity in medical education and may help to develop teaching skills for 3rd year students required to work in the clinic and in 4th year students who take an elective in DOCENT.

Conclusions
Finally, the current clinic logistics offer signi cant opportunity for interprofessional education. Because the clinic is structured, consistent, and frequent, it allows additional professional schools (with different academic calendars, supervisory needs, etc.) to participate. The high percentage of patients with MSK complaints creates an opportunity to involve Doctor of Physical Therapy students and health systemcredentialed ED nurses, creating a further opportunity to involve undergraduate nursing students whose roles are different from the medical students. A future direction of the clinic will be to expand opportunities for interprofessional practice and assess the impact of that collaboration on students' professional development.

Consent for Publication: Not applicable
Availability of Data and Materials: The data that support the ndings of this study are available on request from the corresponding author, AC. The data are not publicly available due to information that could compromise research participant privacy.
Competing Interests: The authors declare that they have no competing interests.
Funding: This research did not receive any speci c grant from funding agencies in the public, commercial, or not-for-pro t sectors.
Authors Contributions: AC conceptualized the study design and methods, provided project administration, and wrote, edited, and reviewed the original draft. EL assisted with conceptualizing the methods, and both writing and reviewing the manuscript. BJT assisted with conceptualizing the methods as well as writing and reviewing the manuscript. YS assisted with the methodology, conducted the data analysis, and reviewed and edited the manuscript. BIP assisted with conceptualizing the methods, and reviewing and writing the manuscript. NH assisted with conceptualizing the methods, and writing, editing, and reviewing the manuscript. AH assisted with conceptualizing the methods, and both reviewing and editing the manuscript. JH assisted with reviewing and editing the manuscript. KW assisted with conceptualizing the methods, writing the original draft, and both reviewing and editing the manuscript. HJL assisted with conceptualizing the methods, conducted data analysis, and assisted with reviewing and editing the manuscript. EB assisted with conceptualization of study design, acquired funding, and assisted with reviewing and editing the manuscript.

12.
Aba Alkhail B. Near-peer-assisted learning (NPAL) in undergraduate medical students and their perception of having medical interns as their near peer teacher. Med Teach. 2015;37 Suppl 1:S33-9. Tables   Due to technical limitations, table 1 is only available as a download in the Supplemental Files section. Figure 1 Satisfaction of DOCENT patients compared to patients who remained in the ED.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. Table1DOCENT.rtf