1) Cognitive Task Analysis
The coded themes and examples are outlined in Table 1. Responses to questions were similar across all stakeholders. Five overarching themes were identified including 1) gathering patient information, 2) filtering and searching for necessary information, 3) subjective, objective, assessment, and plan, 4) visualization of unstructured EHR data, and 5) trends of patient progression and comparisons in graphs.
Coded themes from stakeholder responses to interview questions.
Gathering patient information
“…review current inpatient patients (admits), see if there was any discharge overnight, if so change to outpatient status. Schedule to call discharged patients…”
Filter and search for necessary information
“…Filtering/screening ‘reasons for dx’ or ‘reasons for admit’ would be helpful, especially if could look into PHM, would save time when screening...”
Subjective, objective, assessment, plan of patient
“…look at signs and symptoms, lab values that support the target diagnosis (ex: would look CXR, BNP, EF for CHF). Also look at plan/assessment from doctor, and diagnosis data/info verbiage for COPD CHF, pneumonia.”
Visualization of unstructured EHR data
“The presentation varies, but for daily workflow it would ideally be in a list with patient names, medical record number, location (unit), insurance, etc..”
Patient progress trends and comparisons in graphs (i.e. bar, pie)
“Trends and progression graphs”
For current workflow, all interviewees are required to know patient diagnosis and demographics, insurance status, and side effects of medications. However, it is difficult to find the necessary information from the current EHR due to disjointed information. Instead of manually searching for each piece of information, the workflow allowed a single place to verify, view, and determine patient information.
To make the workflow easier, stakeholders mentioned requiring the ability to quickly vet information through pulling and filtering free text information. Manually reading through every single progress and chart note was time-consuming. Thus, the code for an easier workflow selected was to filter and search for necessary information. Stakeholders mentioned this would save them time and be able to stratify patients, which result in improving their workload.
Some specific information that was required for patient care was subjective, objective, assessment and plan (SOAP). Stakeholders are required to see all aspects of the health of the patient: labs, medications, past medical history, consultations, surgeries, etc. The resulting theme was the SOAP format information. Patient information and treatment plan were necessary to understand each case and make decisions for essential care.
For the visualization questions, stakeholders desired a platform where they could easily see unstructured patient data and also the progress of the patient’s trends. Stakeholders mentioned the current way of displaying patient information was through multiple pages. Instead, they suggested visualizing all patient information in one place with multiple patients as a table format, charts, or graphs.
Finally, intuitive visualizations could aid in stakeholders’ day to day work environment to be effective which allows them to review trends, compare data, and patient progress at a glance. Stakeholders mentioned having a one snapshot view in the visualization dashboard area would be highly beneficial.
2) Health Analytics Dashboard
The developed dashboard included three main functionalities. First, the machine learning algorithm processed unstructured EHR data and parsed it into meaningful information. The visualized information can support clinicians to understand a patient's severity and acuity in one snapshot. Second, the visualization snapshots of patient progress use clear graphics and visual tools to instantly comprehend the patient lab trends and testing results. The AI algorithm can also show a list of patients based on their race or ethnicity. Third, the data from the platform can be downloaded into any file format such as a Microsoft Excel sheet or CSV file.
Figure 3 represents snapshots of the platform. Figure 3a and 3b were created in response to multiple comments regarding visualizing trends and patient’s vitals and lab values in one place instead of checking multiple areas of the EHR to find data. Figure 3a represents vitals such as temperature and blood pressure and Fig. 3b represents lab values such as glucose and hemoglobin. Clinicians and nurses are able to view the trends and make appropriate interventions. For each patient, the trends were depicted as scattered plots connected with lines. Figure 3c represents the solution to that request. In the search bar, clinicians are able to type in certain measures such as blood pressure > 120 to identify patients who have elevated blood pressure requiring intervention. Figure 3c illustrates the search result that found 4 patients who matched and had systolic blood pressure > 120. This could be done with other clinical markers such as lab values, diagnosis, or medications. Figure 3d represents an instance when the clinician searches for patients who take warfarin. The search resulted in categorizing patients who are currently on warfarin and patients who have active problems from taking warfarin. The results are represented in a pie chart with ratios of patients with warfarin and other medications from insulin to atorvastatin. Clinicians were able to view drug-drug interactions and adverse drug events through this resulting page. Some problems with patients on warfarin consist of hypertension, anemia, and active bleeding. Users can also search other medications as well in order to see the impact on the patient population. The visualization utilized the CRNN model to predict a specific patient’s trend and the past trend. For example, if a patient has a blood glucose level of 160 for the last 3 days, based on the ML model, our system can predict blood glucose level in the next 3 days with more than 80% accuracy.
3) SUS Survey
For the SUS survey, participants included clinicians, pharmacy coordinators, nurses, hospital administrators and social case workers. Of the 20 participants, 14 were female and 6 were male. All 20 participants completed the SUS survey. The average raw score was 32.35 and the average final score was 80.9 with a standard deviation of 5.69. The individual scores are graphed as shown in Fig. 4.