Discussion Themes. Table 2 displays the themes, sub-themes, and ideas from the panel discussion. BSL and ACF identified several repeating ideas from the panel discussion. The discussion was divided into two parts: a discussion of the barriers to depression screening and a discussion of the ideas submitted to the innovation tournament.
In terms of the barriers to screening, panelists were concerned about MAs administering the depression screener. The majority of panelists (67%) stated that MAs are not clinically trained to administer a sensitive mental health questionnaire. Most (67%) panelists voiced that the PHQ-2/9 were designed and validated for self-administration. Panelists (67%) also suggested that both clinicians, MAs, and patients may not fully understand the rationale for administering the depression screener. Finally, panelists (33%) indicated that there are several technological challenges related to the way depression screening results are integrated in Epic©.
Panelists discussed the ideas submitted to the innovation tournament. Most panelists (67%) focused on the fact that the “reframing” idea revealed that those submitting ideas to the innovation tournament were less focused on increasing screening rates, but rather improving the accuracy of screens. That is, tournament “reframing” ideas described ways to facilitate a deeper connection between providers and patients; they also recommended changing the wording of the PHQ-2/9 questions for the sake of clarity. The majority of panelists (83%) liked the idea of patient self-report of the depression screener, particularly the implementation of tablet computers in the patient waiting area. Panelists discussed safety and liability concerns with pre-check-in text messages through MyPennMedicine such as the need to ensure that the endorsement of suicidality could be quickly identified and acted upon if the patient is completing the PHQ-9 outside of the office. Most panelists (83%) also liked the idea of patient education and thought it was necessary, though one panelist voiced that patient education would unlikely increase rates; rather it would improve the accuracy of the depression assessment. Panelists felt that both ideas would also be feasible and acceptable to implement in the practices.
Election Results. The instant runoff election winner to the rank-choice election was the patient self-report idea. The panel determined that the ideas that suggested patients complete the depression screener confidentially either on tablet computers, through MyPennMedicine ahead of the visit, or via text message were the most viable and potentially impactful strategies to improve depression screening.
Phase 3 — Piloting the Winning Innovation Tournament Strategy
Overview of Rapid Prototyping
Rapid prototyping is the systematic testing of ideas in order to create and refine strategies quickly (50). First employed in industrial design, this method has been extended to healthcare contexts where effective implementation strategies don’t yet exist and immediate feedback is necessary to optimize healthcare quality and safety (38,51–55). Rapid prototyping facilitates learning as quickly as possible whether a strategy works and allows researchers to make adjustments as needed. Identified problems are documented and the implementation plan is revised. A subsequent experiment is conducted to see if it resolves the problem and to identify any further problems. Rapid prototyping is done iteratively and cyclically, much like Plan-Do-Study-Act cycles frequently used in quality improvement studies (56). See the analysis plan below for the process description.
Method
Participants. To evaluate the feasibility and acceptability of the winning innovation tournament idea, we planned to first pilot in two PIC primary care practices before scaling the project up. The research team met with leadership in the Primary Care Service Line, including leaders from each PIC practice, and two practices agreed to participate in the pilot. Along with an informational technological consultant from the health system, two members of the research team met with the practice manager and the lead clinician of each practice to discuss the pilot workflow changes. Given that piloting the strategy would involve substantial changes to the workflow, one practice requested to initially pilot the tablets with one physician who already adhered to depression screening guidelines before scaling to the entire practice. Due to COVID-19 pandemic related physical distancing precautions, the pilot was halted after prototyping in one practice with one physician.
Procedure
Study Design. Data collection was conducted according to a withdrawal design method—i.e., a method in which the intervention is “withdrawn” systematically to allow for a comparison between changes in the outcome in baseline versus intervention periods (57). The research team was present on specific “intervention” days to use tablets for depression screening, and on alternating “baseline” days, depression screening was conducted as it was normally conducted in the practice.
In the case of this particular PIC practice, usual care for depression screening involves the MAs administering the PHQ-2 to patients verbally. MAs enter patients’ responses in Epic©. If the patient screens positive on the PHQ-2 (a score of > 2) MAs provide a paper version of the PHQ-9 for the patient to complete; the patient is then expected to hand off the completed questionnaire to their clinician. Once the clinician arrives in the examination office, the clinician is expected to manually enter the depression screening data in Epic©.
Materials. Two tablet computers were purchased to conduct the rapid prototyping. To ensure patient safety and confidentiality, the health system encrypted each tablet and installed Epic Welcome©, the patient-facing application version of Epic©. Patients complete questionnaires and consent forms on Epic Welcome© and their responses sync in real-time with the clinician-facing version of Epic©. On Epic Welcome©, when patients complete the PHQ-2, the questionnaire automatically expands into the PHQ-9 if patients’ score on the PHQ-2 is greater than or equal to 1 (endorsing at least some symptoms on either of the two items). Notably, this is a more liberal cut-off than CMS and health system guidelines, which recommend follow-up at a PHQ-2 score greater than 2. This discrepancy provides further evidence that significant technological barriers prevent health system standardization of depression screening, as described by innovation tournament participants and panelists.
Rapid Prototyping Analysis Plan
Rapid prototyping was conducted over five days (i.e., five 5-hour shifts) in a PIC practice. The three steps of the rapid prototyping process are outlined below.
Step 1 — Design. Before each 5-hour shift, members of the research team (BSL and ACF) designed a plan to use tablet computers to screen for depression. This plan included decisions about the workflow, materials needed, the staff involved in the process, and the location of screening. To ensure rapid feedback, the research team collected field notes and interviewed stakeholders (PSAs, the practice manager, MAs, the physician, and patients) involved in the pilot. Field note templates and qualitative interview guides are provided in Additional File 1 and Additional File 2 respectively. Qualitative interviews were transcribed verbatim in real-time. The research team also documented whether the PHQ-2 was administered.
Step 2— Evaluate and Review. Immediately after a 5-hour shift of rapid prototyping, the research team met to review the findings. Field notes and interviews were read aloud together and synthesized to eliminate redundancies and ascertain discrepancies. Qualitative data were analyzed using a rapid immersion/crystallization approach (36,58–60). BSL and ACF, who had been extensively immersed in the experience, developed impressionistic summaries of what they learned. Researchers’ holistic impressions of the experience were crystallized through discussion and written documentation. To ensure systematicity, the research team also recorded key features of each rapid prototyping cycle: (1) a summary of the workflow design; (2) workflow successes; (3) workflow challenges; and (4) a summary of the changes to be tested in the next cycle.
Step 3 — Refine and Iterate. After determining the necessary workflow changes, the research team planned to refine the tablet screening process. This process sometimes involved writing scripts for the PSAs presenting tablets to patients to ensure that patients received uniform rationale about the PHQ-2. Other times, this involved placing laminated sheets with screenshots to guide MAs and the clinician to find the PHQ-2 depression screening data in their version of the electronic health record. The research team communicated with the practice ahead of the shift to ensure the changes were acceptable. The new iteration was then tested in the subsequent shift. In order to be able to directly evaluate whether the specific iteration of the strategy was superior to the previous cycle’s, attributes were not modified if they did not present challenges. This process repeated for each cycle.
Outcomes
Qualitative Data. Field notes and interview transcripts from each cycle of the rapid prototyping process were collected and analyzed to iteratively improve the pilot process.
Quantitative Data. The primary outcomes for the pilot were PHQ-2 and PHQ-9 screening rates and PHQ-2 follow-up. According to CMS in 2019, follow-up to the PHQ-2 (if the score is greater than 2) is considered complete if one of the following actions is taken: (1) the patient completes a more extended depression questionnaire (i.e., PHQ-9) or suicide assessment; or (2) the primary care clinician refers the patient to a mental health clinician; or (3) the clinician prescribes depression medications; or (4) the clinician documents a depression follow-up plan; or (5) the clinician documents a depression diagnosis.
Depression screening data were extracted from Epic©. Patient eligibility for the screener, PHQ-2 scores, PHQ-9 scores, medication list, medical diagnoses, referrals, and patient notes were extracted from Epic© for intervention and baseline shifts. To ensure a fair comparison and avoid any potential confounds related to the timing of the visit, baseline shift data were collected from the same time window and same physician as the intervention shifts. Because tablets automatically triggered the PHQ-9 at a lower score (a PHQ-2 score > 1) than CMS requirements and the usual care practice (a PHQ-2 score >2), PHQ-9 and follow-up rates were compared statistically based on CMS requirements (PHQ-2 score > 2).
Results
Qualitative Results. Table 3 displays the rapid prototyping process results from each day of piloting; Additional File 3 narratively describes these detailed results. Figure 2 displays the modifications of each rapid prototyping cycle and tablet administration results.
Broadly, rapid prototyping revealed that tablet computers were acceptable and feasible to most stakeholders (the physician, the practice manager, MAs, and patients) involved in depression screening. However, PSAs found the additional responsibilities of checking patients in, handing off the tablets, and introducing the screener to patients to be somewhat disruptive to their workflow. In addition, anywhere between 1-3 patients during the first four five-hour shifts did not complete the questionnaire via tablet either due to the patient’s physical limitations, because the workflow was not yet mastered by stakeholders, or because there was not sufficient time to complete the depression screener via tablet. These patients completed the PHQ-2 verbally with the MA (i.e., through usual care). After several refinements were made during the rapid prototyping process, the process of depression screening via tablet improved and the PSAs eventually described that the workflow adjustments were “smooth sailing.” During the final shift, all patients were screened via tablet and stakeholders considered the pilot a “success.” Stakeholders made several recommendations for how to scale up the process across PennMedicine primary care practices, including transformations to Epic© and standardization of depression screening workflows across the health system.
Quantitative Results. Overall, the rapid prototyping process resulted in comparable PHQ-2 screening rates to usual care (given that the practice participating in the pilot already screened all eligible patients). However, tablets significantly increased PHQ-9 screening rates. Follow-up between usual care and piloting days were comparable. Z test results for non-significant findings should be viewed in light of the fact that the pilot study was not adequately powered to detect a statistical difference.
On baseline days 30 out of 30 (100%) eligible patients were administered the PHQ-2. On intervention days 35 out of 36 (97%) of eligible patients were administered the PHQ-2. According to field notes, the patient whose PHQ-2 data were not recorded was given a tablet, indicating that responses were not synced with Epic©. Given that the practice participating in the pilot was highly compliant with PHQ-2 screening guidelines, differences between baseline and intervention days were not significant (z = 0.92, p = 0.36).
For those patients whose PHQ-2 score was positive (a score >2), PHQ-9 screening was evaluated. On baseline days, 1 out of 3 (33%) of eligible patients were administered the PHQ-9. On intervention days, 6 out of 6 (100%) of eligible patients were administered the PHQ-9. Differences between baseline and intervention days were significant (z = -2.27, p = 0.02).
Using the more inclusive definition of PHQ-2 follow-up per CMS guidelines (described above), follow-up rates were extracted from Epic©. On baseline days, 2 out of 3 patients (67%) received follow-up after a positive PHQ-2 screen. On intervention days, 6 out of 6 patients (100%) received follow-up after a positive PHQ-2 screen. Differences between baseline and intervention days were not significant (z = -1.50, p = 0.13).