User experience of a family health history chatbot: A quantitative analysis

Objective: Family health history (FHx) is an important tool in assessing one’s risk towards specific health conditions. However, user experience of FHx collection tools is rarely studied. ItRunsInMyFamily.com (ItRuns) was developed to assess FHx and hereditary cancer risk. This study reports a quantitative user experience analysis of ItRuns. Methods: We conducted a public health campaign in November 2019 to promote FHx collection using ItRuns. We used software telemetry to quantify abandonment and time spent on ItRuns to identify user behaviors and potential areas of improvement. Results: Of 11065 users who started the ItRuns assessment, 4305 (38.91%) reached the final step to receive recommendations about hereditary cancer risk. Highest abandonment rates were during Introduction (32.82%), Invite Friends (29.03%), and Family Cancer History (12.03%) subflows. Median time to complete the assessment was 636 seconds. Users spent the highest median time on Proband Cancer History (124.00 seconds) and Family Cancer History (119.00 seconds) subflows. Search list questions took the longest to complete (median 19.50 seconds), followed by free text email input (15.00 seconds). Conclusion: Knowledge of objective user behaviors at a large scale and factors impacting optimal user experience will help enhance the ItRuns workflow and improve future FHx collection.


Introduction
Family health history (FHx) is an important tool for assessing a patient's risks of speci c health conditions such as cancer. Knowledge of FHx could help healthcare professionals determine the potential risks for an individual and recommend targeted intervention and preventive measures. Web-based, patient-facing tools have been developed to systematically collect and organize FHx outside of the provider's o ce, improving e ciency of clinical visits while increasing completeness and accuracy. [1][2][3][4] These tools allow users to generate pedigree, collaborate with family, provide decision support, and support health information technology (HIT) integration.
Automated conversational agents (chatbots) have emerged as a promising approach for health data collection. Chatbots promote a friendly, intuitive, and engaging user experience. 5,6 One such effort is ItRunsInMyFamily.com (ItRuns) -a web-based, simple, and scalable tool that engages users in a conversation using a chat interface to collect FHx and provide recommendations regarding hereditary cancer risks. 3 Research has demonstrated data collection chatbots (such as ItRuns) reduce the workload on users and are perceived as more usable and satisfactory compared to standard web-based forms, even when collecting granular FHx. 6 Collection of FHx can involve detailed questions and signi cant time investment, making it imperative to provide users with an optimal and engaging experience in using FHx tools. Very few studies have explored the user experience of these tools. 1 Research has found that FHx collection tools can produce poor user satisfaction with respect to assessment format, data-input method, and complex clinical jargon. 2 Patients' incomplete knowledge of family members' health history and con dentiality concerns remain as barriers to FHx tool adoption. 7,8 In addition, factors such as time taken to collect comprehensive FHx and types and amount of data inputs could also lead to under-utilization and poor user experience.
Research has focused on understanding user experience and preferences employing qualitative and experimental approaches such as surveys, interviews, comparative assessments, and randomized control trials. 2-4, 6,9 The purpose of this study is to explore quantitative software telemetry (collection of software usage information) to assess granular, objective user behaviors, abandonment (where and when a user leaves the FHx tool) trends, and time to complete various steps to identify potential pitfalls and areas of user experience improvement. 10 To the best of our knowledge, no studies have explored user behaviors of completing FHx assessments using telemetry data. In this paper, we explore the software telemetry to assess user behaviors at a large scale and identify potential factors impacting user experience. The insights gathered from this quantitative evaluation will allow us to enhance the overall user experience of ItRuns assessment.

ItRunsInMyFamily.com and November 2019 10K Campaign
ItRuns is a web based FHx collection and hereditary cancer risk assessment tool developed by researchers at the Medical University of South Carolina and ItRunsInMyFamily.com LLC. ItRuns is a free, secure, browser-based, and mobile-rst application that does not require the user to download software or to create an account. 3 ItRuns uses a dialogue-based text chat interface to mimic human-to-human conversation providing its users a natural and engaging experience analogous to talking with providers.
The current version of ItRuns uses an innovative chatbot interface that simulates a natural conversational dialogue to collect FHx from users. The chatbot engages users in a structured and intuitive way, rather than multiple pages of online forms, tables, or a series of complex questions compared to other tools. Upon completion of the ItRuns assessments, a personalized risk assessment report (section 3.2.1) is sent to the user's email address. 3 This paper reports on data collected from an online FHx collection campaign conducted using ItRuns. 11 This campaign was conducted between November 1, 2019 and November 30, 2019 in conjunction with Family Health History Month. 11 The ItRuns team aimed to recruit at least 10,000 English-speaking users via paid and unpaid marketing campaigns through social media promotion, contacting previous users of ItRuns, using promotional banners on the ItRuns website, and direct contact with cancer support groups. Interested users completed the assessment using the ItRuns assessment link on mobile devices or computers. Users consented to take the ItRuns assessment by clicking on consent checkboxes within the assessment. This secondary analysis study was deemed not human subject research by the Medical University of South Carolina Institutional Review Board (Pro00094990).

ItRunsInMyFamily Work ow
This section provides a step-by-step description of the ItRuns assessment work ow divided into 13 sub ows (SF) as explained below (See Fig. 1). SF 1 | Introduction: The assessment begins with an introduction to Dokbot -the chatbot persona users engage with during the assessment. 12 Dokbot introduces the user to the purpose of ItRuns and obtains informed consent. SF 2 | Basic Demographics: Upon consent, Dokbot collects the user's name and gender. Dokbot also asks if the user is 18 years or older. If yes, Dokbot continues with the assessment. If not, the assessment ends with a message explaining that users have to be at least 18 years of age to take the assessment. SF 5 | Family Cancer History: Dokbot then asks if anyone in the family has had cancer. If the user reports yes, Dokbot asks which relative, their name, cancer, age of diagnosis, as well as cancer-speci c follow up questions, if the relative had any other cancers, if they are still alive, their current age or age when they died, and whether the family member has ever been tested for a genetic mutation in a cancer-causing gene. This sequence is repeated for each relative the user reports as having been diagnosed with cancer. SF 6 | Pedigree: Dokbot asks the total number of daughters, sons, sisters, brothers, maternal aunts, maternal uncles, paternal aunts, paternal uncles in their family. Numbers of nieces and nephews were not assessed due to the complexity of the data and its limited utility. For female users, age when they had their rst child is assessed during this sequence. In this manner, a 3-generation pedigree is collected. SF 7 | Physical Traits: Dokbot collects users' age, weight, and ancestral origin. If users indicate that they are of European descent, Dokbot asks whether they have Ashkenazi Jewish ancestry as these individuals are at a higher risk of BRCA gene mutation. 13 SF 8 | Female Details: If the user is a female (indicated in SF 2), they are asked the age at menarche; if they have had their breasts, ovaries, or uterus removed, if they have gone through menopause and at what age; if they take hormone replacement therapy if they had a mammogram. If they have had a mammogram, Dokbot asks the date of the last mammogram and the history of dense breasts. They are also asked if they've had a breast biopsy, if yes, number of biopsies, any abnormal ndings and type of abnormality. SF 9 | Colorectal Cancer Screen: Dokbot asks all users about the history of colonoscopy. If yes, Dokbot further asks if polyps were found and, if so, the number and type. SF 10 | Lifestyle: Dokbot collects a user's zip code, smoking history, tobacco use and alcohol use. If the user reports having smoked cigarettes, Dokbot asks for their age when they started smoking and the age when they stopped; if they report they are no longer a current smoker, what number of cigarettes per day when they did smoke. Dokbot also collects ibuprofen and aspirin use. SF 11 | Thank you and Email: Dokbot thanks the user for participation and collects the user's email to send the ItRuns risk analysis report. SF 12 | Recommendations and Report: Dokbot providers high-level risk recommendation based on family health history and guidelines for hereditary cancer risks. Dokbot also informs the user that a detailed report (section 2.4.1) will be shared via email.

SF 13 | Invite Friends: Users have an option to share ItRuns assessment with friends and relatives on
Facebook by posting on their public pro le.

ItRuns Risk Assessment Report
ItRuns uses ontologies, ontological reasoners, clinical practice guidelines, and web services to provide evidence-based recommendations to users based on their FHx. Owlready2 ontology module and Protégé ontology platform were used to develop a lightweight, patient-centric clinical practice guideline domain ontology using hereditary cancer criteria from the American College of Medical Genetics and Genomics and the National Cancer Comprehensive Network. The development of this ontology-driven clinical practice guideline criteria risk assessment is published in a separate publication. 14 A risk analysis PDF report is emailed to participants who completed the ItRuns assessment. The report contains an Executive Summary, followed by a Guidelines section consisting of the hereditary cancer predisposition criteria the user meets and the published recommendations for cancer predisposition assessment they should follow. Next, the Health History section includes information about the user and their health history (including a family pedigree), a breakdown of relatives with cancer, relatives who are also at risk, and family cancer statistics for the user's family. Finally, the Recommendations section includes additional details about the hereditary cancer syndrome the user for which the user might be at risk, available genetic counseling resources, and additional genetic testing information. The About section includes content about the product, contact information, support, references, and a legal disclaimer.

Study Participants
Potential users were invited via paid and unpaid campaigns as described in Section 2.2. They provided consent to participate in the study at the beginning of the assessment. Adult users (≥ 18 years) were allowed to continue with the assessment after providing consent. Users were asked for their email address to send their Cancer Risk Assessment report at the end of the assessment. No other personal identifying information was collected.

Telemetry Data
During the assessment, the Dokbot software collected and stored in a relational database telemetry data detailing the user's behavior interacting with the software, including timestamps of all actions, transitions between steps, geographical location, referral URL, browser information, and supported languages.

Data Analysis
Once the campaign was completed, telemetry data was extracted from the Dokbot database. A script was run to remove small amounts of unusable data, including records where a user loaded the ItRuns work ow but did not submit any single step as well as researcher/developer test records. Then, we extracted elements such as unique random user ID, step description, step start and end timestamps, etc. to assess the experience.
Using an analysis script, we quanti ed the telemetry data to calculate the usage, abandonment rate, and time per step. We tabulated and graphically represented this data using Microsoft Excel software. Descriptive measures were used to obtain the frequencies, percentages, mean, and median related to completion, abandonment rate, and time taken for each sub ow and related steps.

Results
Between the testing period, 14,140 users clicked on our marketing campaign and landed on the ItRuns homepage. Here, we report on usage behaviors of 11,065 (78.25%) users who started the ItRuns assessment. A total of 4,305 (38.91%) users completed the assessment and reached the nal step to receive a recommendation. About 575,000 Dokbot steps were presented. On average, highest departures or assessment abandonments were in the Introduction (32.82%), Invite Friends (29.03%), and Family Cancer History (12.03%) sub ows. Most (89.20%) individuals left the assessment without inviting family members or friends to take the ItRuns assessment. Table 1 summarizes the key steps in the assessment work ow including departures per step, percentage abandonment, and time spent to complete the steps. The third largest overall drop was seen at the Family Cancer History sub ow (Fig. 4). Users may have to enter cancer history details of one or more family members (e.g., mother, father, etc.), so the user may complete all or some of the steps in this sub ow multiple times. In this study, we did not determine when the user abandoned the assessment if they completed the sub ow multiple times. The highest (5.26%, 43.73% of the sub ow) abandonment was seen during the step when users were asked about which of their family members had cancer followed by relative's names (2.89%) and types of cancer they had (2.08%).

Abandonment by steps
We looked at abandonment by individual steps as opposed to overall abandonment to identify questions that are particularly challenging to users, despite only being seen by a small percentage of users. Table 2 provides details of the top steps of abandonment.

Time per step
The median time to complete the assessment was 636 seconds or 10.60 minutes. Table 1 demonstrates the total median time spent on each sub ow. Users spent the highest median time to complete the Proband Cancer History (124.00 seconds) followed by Family Cancer History (119.00 seconds), Female Details (67.00), Pedigree (55.00 seconds), and Lifestyle (53.00 seconds). Users spent the least amount of time in Basic Demographics (7.00 seconds), Lifestyle (8.00 seconds), and Invite Friends (9.00 seconds) sub ows.   Table 4 for more details.

Discussion
We sought to understand the user experience of ItRuns to enhance FHx collection. This paper quantitatively explores software telemetry data to identify actual user behaviors and actions using software telemetry. Our ndings suggest that factors such as lack of knowledge or interest in FHx, trust in FHx tools, data sharing and con dentiality concerns about sharing personal and family health information, lack of collaboration with family members, and increased time spent on conversational textbased work ow can contribute to suboptimal user experience. We discuss potential considerations for improving user experience of FHx tools and implications for ItRuns.

Main ndings
About one in three users (32.81%) abandoned the assessment in the initial Introduction sub ow with a 22.03% drop after the greeting statement. We hypothesize that potential reasons for this drop could be the lack of knowledge or interest in cancer risk assessment and distrust or privacy concerns in using a casual, informal, chat-based agent to share personal and family health information. In a previous study, users expressed concerns regarding con dentiality of sharing FHx using digital tools. 7 In healthcare chatbot applications, lower reading level and informal language is sometimes perceived to be less trustworthy. 15 We observed abandonment when asked to provide their consent to begin assessment. While ItRuns ensures users that their data is protected with compliance regulations (including HIPAA, GDPR, and CCPA), concerns over sharing personal health information with a third-party solution (not directly recommended by their provider or as a part of a marketing campaign) could restrict users from moving forward. Research shows that while patients are comfortable with sharing personal health information with their own providers, they often restrict sharing information outside of their care network. 16 Over 12% of user drop was seen in the Family Cancer History sub ow. Commonly, users abandoned the assessment when asked about which relatives had cancer, their names, and types of cancer they had. As seen in previous research, users may have abandoned at this stage of the assessment if they did not have the correct or complete information about family cancer history. 4,17 As ItRuns currently does not allow live collaboration with family members or return to complete assessment at a later time, users could abandon the assessment considering the potential lack of speci c family health information. Also, the distrust of the technology and skepticism to share personal health information of family members with ItRuns (a third-party solution) could also play a role. In addition, the looping nature of this work ow required users to add one or more family members' information one-at-a-time and with one or more occurrences of adding cancer and genetic testing information. About 7,000 users completed some steps of these work ows over 22,000 times (average of three repetitions per user). This repetitive nature of ow took an average of 119 seconds to complete with the top steps averaging 53.00 seconds.
A high (23.90%) drop was seen at the end of the work ow when users were asked if they would like to share ItRuns assessment with family members and friends via Facebook. While waiting for their risk assessment results, the assessment asked, "While you are waiting on your report, will you invite your friends and family to use ItRunsInMyFamily so they can learn their family cancer risk as well?". Research suggests that users may not be interested in sharing health information online but rather personally via direct communication. 18 People tend to be cautious of sharing health information online, as few formal policies exist to ensure protection of users' information and the ease of which aggregated health information can be re-identi ed. [19][20][21] In future, development of approaches such as sharing with family and friends personally via emails or text message could help better promote ItRuns and spread awareness about FHx collection.
Users took about 636 seconds (10.60 minutes) to complete the assessment. This time is lower compared to reported times for FHx collection in-person or using digital health tools (average 15 to 60 minutes). 1,7,8,22 In a previous study, the previous version of ItRuns took a higher time compared to standard FHx web-based form. 6 Our initial hypothesis was that ItRuns will require a high completion time resulting in increased abandonment. As ItRuns uses a conversational text message-like interface, users completed certain questions (types of cancer and genetic tests, family member details) multiple times, not aware of these repetitions at the beginning, able to see all information at once, or recollect previously added information without scrolling up the chat history. Also, in the version, users were unable to change any incorrect responses. was the 4th most common option users chose. 23 As previously reported, the position of the 'Other Cancer' option and lack of cancer synonyms (uterine cancer instead of endometrial cancer) could have led to selection of this choice. 23 Also, if users misspelled a cancer, they might not see the correct option due to the lters. These factors could have led to higher time spent on these sub ows. Users also spent about 15 seconds on the question asking for their email address. It is possible that users may have taken time to decide which email address to provide and type it in the text eld, which is time consuming. Although, ItRun's privacy statement ensures users that their personal information will not be shared, an initial hesitation to submit personal email address in a marketing campaign could have resulted in a longer response time. We saw a total of 6.69% abandonment in two text input questions asking for user's name or family member names, which may likely be due to hesitation to share their own family members' personal information.

Limitations
The majority of participants were women between the ages 40 and 60 years. The ndings may not be representative of the general population. Although, research shows no noticeable age and gender group differences in technology literacy compared to our population. 24 We aimed to recruit a large sample (over 10,000 individuals) employing a wide cast marketing strategy.
This could have resulted in individuals completing the assessment inaccurately or without genuine interest and providing fake data. It is possible that individuals may have completed the assessment in a hurry or taken additional time to complete the assessment at their own convenience. In future research, we aim to recruit a representative clinical population to validate these ndings.
The lack of collaboration capabilities for adding personal or family health history within ItRuns could have led to increased times for completing the assessments. Allowing ItRuns users to collaborate with family members and the ability to interact with personal or electronic health records could promote better experience and engagement.
ItRuns collects information in a conversational chat-based approach potentially resulting in higher time to complete the assessment. However, research shows that users prefer the chat-based experience compared to standard, web-based data collection approaches. 6 The analyses presented in this study were conducted with data extracted from a large-scale software database. The validity and quality of large data sets can vary substantially. 25 It is particularly important to validate minimum database quality in healthcare contexts. 26 Our team completed error checks and validations to ensure data integrity.

Future work and implications
This quantitative exploration has future research implications. To the best of our knowledge, we are one of the rst teams to conduct a FHx public health campaign to recruit a large-scale sample of over 10,000 individuals and contribute to user experience assessment by analyzing software telemetry data.
Based on this research, we recommend the following considerations to enhance the user experience of end user facing FHx tools and describe how we will improve ItRuns in future.
Promote FHx education by embedding education and resources within FHx tools to help users with decision making. We have already developed features that would allow us to embed tailored education material within Dokbot, such as importance of FHx, hereditary cancer risks, and role of genetic counseling; resources to connect with public health agencies to learn more about these topics, etc. as well as assure users about their information being secure within ItRuns to promote its use without concerns and bridge the potential knowledge gap about FHx collection.
Build and maintain trust while collecting FHx by establishing strategies (such as positive reinforcement messages) to alleviate privacy and con dentiality concerns. Although ItRuns informs users their data will be used for research and not for marketing purposes, we will reassure users when asking about personal and family health history and other identi able information. We will assure users that ItRuns and Dokbot are compliant to privacy regulations. Patients often trust data sharing recommendations from their providers. 16 ItRuns can be embedded within patient portals and patient management systems such that providers can directly prescribe it to patients.
Empower users and reduce burden by promoting collaboration between family members to gather complete and accurate FHx, reduce FHx collection time, increase adoption, and bridge the knowledge gap. We aim to allow ItRuns users to collaborate and share assessment with family members and save progress to continue the assessment at a later time without losing previous responses (Save and Return feature). We have implemented features to provide information on progress and time estimates to keep users informed and motivated.
In addition, future research could also address the following: This study did not directly study the user's preferences, but rather via their software usage and actions. Future studies could supplement such analyses with user interviews or focus groups to better understand their experience, perceptions towards ItRuns, and test the hypotheses for assessment abandonment at various stages.
This study did not aim to assess the quality of data gathered during this study. Future researchers should also consider assessing the accuracy and completeness of the data collected using the chatbot interface.
Lastly, our approach could be readily applied to other healthcare assessments to evaluate user experience and engagement. As a part of this research, we have developed a generalizable program script which could be used to extract and summarize telemetry data of assessments developed using chatbot application-Dokbot. 12 We aim to continue studying and improving on other custom and standardized Dokbot assessments to enhance patient and provider experience of healthcare data collection.

Conclusion
This paper quantitatively explores the user experience of ItRunsInMyFamily.com FHx assessment. We used a unique approach by employing software telemetry to assess actual user behaviors related to abandonments and time spent on various ItRuns Steps. The ndings provide insights on factors impacting user experience and assessment completions. Key factors include the potential lack of knowledge or interest in FHx, lack of trust in FHx tools, con dentiality concerns about sharing personal and family health information, lack of collaboration with family members, and increased time due to conversational text-based work ow. We discuss considerations for improving FHx collection experience and how our ndings will help in the development of the future ItRuns versions for better user experience and engagement.
Declarations Figure 1 ItRuns assessment work ow Percentage of retained users by sub ows Figure 3