Smartphone-based Mobile Applications for Adverse Drug Reactions Reporting: global status and impact on the quality and quantity of individual case safety reports


 Background: Smartphone technology can support paperless reporting of adverse drug reactions (ADRs). The aims of this study were to systematically assess smartphone ADR-reporting applications, understand their qualitative and quantitative impact on ADR reporting, and garner key lessons from owners and developers.Methods: This study had three components: (1) An assessment of ADR-reporting apps, (2) an online survey on the impact of app implementation on ADR reporting and the experiences of app developers and owners, and (3) a search of VigiBase, the World Health Organization global database of individual case safety reports (ICSRs), to observe trends in the number of ADR reports targeting countries where the apps were implemented. Results: Twenty-two apps were included. Most (8/22) were for countries in the WHO African region. Features observed included E2B data elements and functions supporting reporting and user engagement. Seventeen app developers and owners answered to the survey and reported overall positive experiences with app features, and post-launch increases in the total number of ICSRs. User type and user environment were cited as factors influencing app use: Respondents said younger people and/or those with an inclination to use technology were more likely to use apps compared to older or more technology-averse people, while respondents in countries with limited internet connectivity reported persistent difficulties in app use.Conclusions: Smartphone apps for reporting ADRs offer added value compared to conventional reporting tools. Reporting tools should be selected based on interface features and factors that may influence app usage.

Apps available only in languages other than English were translated into English during data extraction.

Online survey
The online survey was designed to interview developers and owners of the selected apps to obtain information on the impact of app implementation on the quality and quantity of ADR reports, and to understand their experience of the apps. The online survey was created using Google Forms and an invitation was emailed to the developers and owners of the selected apps. Up to two reminders were sent to those who did not respond. The survey consisted of 9 open-ended and 10 closed questions (Appendix 2).
The apps' quantitative impact was assessed using responses to questions on total number of downloads, trends in the overall number of ADR reports after the app launch, total number of ADR reports received from the app, and proportion of ADR reports received from the app among all reports since the launch. The total numbers of ADR reports received from the app were also averaged by year based on the number of years from app launch to the survey. The collected responses on the number of downloads of different versions of the Med Safety app were con rmed by MHRA, which had access to such information.
The apps' qualitative impact was assessed using responses to questions on the availability of the minimum four information items for a valid safety report as de ned by the ICH: Identi able patient, identi able reporter, adverse event/reaction, and suspect or interacting drug [10]. The respondents were invited to select those of the four that were judged to have been appropriately lled in by the reporters of the ADR reports received through paper forms and apps.
The lessons learned from the respondents' experiences of the app were sought through open-ended questions. The feedback was reviewed and synthesized into the following themes: simplicity of use, report quality, accessibility, innovativeness, data transferability and data sharing, two-way communication, cost, and data security.
Additional information-such as the app's launch date, the purpose of its implementation, and its development and maintenance costs-was also obtained.

Quantitative search of number of reports submitted to VigiBase before and after app launch
VigiBase was searched to observe trends in the number of ADR reports submitted before and after an app's launch. VigiLyze, the analytical platform of VigiBase, was used for this analysis. VigiLyze did not allow searches by type of reporting tool. Thus, only the overall number of ADR reports, regardless of type of reporting tool, was extracted according to the following search conditions: Geographical scope: Countries where the selected apps were implemented Timeframe: The pre-launch period was de ned as the 12 months preceding the app's launch Post-launch Period A was de ned as the period from the launch to Month 12 after the launch Post-launch Period B was de ned as the period from Month 13 to Month 24 after the launch Although the numbers of ADR reports were not analyzed by type of reporting tool, this analysis was considered to illustrate the quantitative impact of app implementation in the countries.
The VigiBase search was performed on February 15, 2021. Only apps that had already been launched for at least one year (12 months) by that date were included in the analysis, as well as those that had been launched for at least two years (24 months) so that, at a minimum, data for Period A was available, and in some cases also data for Period B. Since the launch date was required to set the timeframes to be analyzed, an app was only included if the online survey response supplied its launch date. Relative percentage changes were calculated to compare the numbers of ADR reports received in post-launch Periods A and B with the number in the pre-launch period for each app.

Search results and app inclusion
A owchart of the app selection is shown in Figure 1. The searches in the Google Play Store and App Store resulted in the retrieval of 4,126 and 1,359 apps, respectively. The number of hits varied depending on the search terms, from 181 hits for 'ADR reporting' to 250 hits for each of nine other terms in the Google Play Store, and from four hits for 'ADR reporting' to 209 hits for 'drug safety report' in the App Store. After removing 2,681 duplicates, 2,144 and 660 apps, from each store respectively, were included in the rst screening stage during which the names and descriptions of apps were reviewed.
Following the rst screening stage, 2,110 apps retrieved from the Google Play Store and 649 apps from the App Store were excluded as they did not meet the selection criteria. This resulted in the retrieval of 34 apps from the Google Play Store and 11 apps from the App Store for the second screening stage (full screening). Thirty apps were excluded in the full screening for the following reasons: "Duplicate" (n=3), "Dysfunctional ADR reporting system" (n=3), "No ADR reporting system" (n=6), "Report submission not automated" (n=13), "Unable to log into app" (n=4) and "Unidenti able app" (n=1). No additional apps were identi ed during the desk research using the Chrome and Firefox browsers.
One of the identi ed reporting apps, Med Safety, had several versions, each adapted to the ADR reporting system of each of eight countries. To assess the versions separately and to facilitate the data extraction process, each of the eight was considered an individual app. Consequently, the nal number of apps that met the inclusion criteria was 22.

Basic and geographic speci cations
The basic speci cations of the 22 selected apps are listed in Table 1. No app required payment by users for download and use. The selected apps were developed by 12 developers, and the MHRA was cited as a developer for 11 apps, such as HALMED, Med Safety, UAE RADR and Yellow Card Scheme. Six of the 22 apps were available only on Android, one only on iOS, and 15 on both platforms. The last update occurred in the study year (i.e. 2020) for 14 apps, one year before the study year (2019) for three apps, two years before (2018) for two apps, three years before (2017) for two apps, and four years before (2016) for one app. Where data is different between Android and iOS, they are shown separately for each platform. When the update dates retrieved from the website con icted with those in the questionnaire, the answer from the questionnaire was prioritized. a: The same app was named Yellow Card -MHRA in iOS.  (20/22), while languages other than English, such as French, were also included in nine apps. Two apps from Argentina and the Russian Federation used only their own national language, respectively. When we classi ed the apps by country income level, as de ned by the World Bank [33], more than 70% of the selected apps were in LMICs.

E2B data elements
The E2B data elements that were available in the selected 22 apps are listed in Table 3. With the exception of one app (Salamtok, developed for reporting in Yemen), where only limited free-text elds were available, all apps consisted of E2B data elements, ensuring that at least the minimum required information could be collected. Patient name or initials were the most commonly adopted elds (21/22) for patient identi cation, followed by elds related to age and gender (20/22). To identify the reporter, a eld for the reporter's name was included in most of the apps (20/22). A eld to describe the adverse reaction or select the reaction term from a de ned list was available in 21 of the apps, of which three did not have further elds, such as date of reaction onset, outcome of reaction and seriousness criteria of reaction. A eld to describe or select drug names from a de ned list was available in 21 of the apps, of which two did not have further elds to add further details about the suspect drug, such as dosage, indication and route of administration. • Only available free-text elds were labeled "City" and "Description" • Only available free-text elds were labeled "City" and "Description" • Only available free-text elds were labeled "City" and "Description"

Additional apps features
The available features that each selected app offered are shown in Figure 2. The features that contributed to ADR reporting included providing support to create a report, report management, and two-way communication, such as not only enabling users to report but also providing drug safety information.
Most of the apps enhanced reporting and two-way communication through a feature for o ine report development (17/22), provision of contact addresses for inquiries (18/22) and provision of drug safety information (15/22). Apps rarely had an automated feature to allow a reporter to share the report with their physician or other addressees of their choice (1/22).

Online Survey
The developers and owners of the selected 22 apps were invited to share their experiences of the apps through the online survey. Answers for 17 of the 22 apps (77.2%) were collected in November and December 2020. No responses were received from the following ve apps: ADR PvPI, Easypharm, Medicinal Vigilance, Salamtok and TMDA Adverse Reactions Reporting Tool.
All of the apps were launched after 2014 except ADR Online (developed for reporting in New Zealand), which was launched in 2010. The respondents con rmed that all the apps were developed to offer an additional tool for ADR reporting besides conventional methods. It was stated that the apps' implementation was also intended to provide users with drug safety information and to raise awareness of ADR reporting through the engagement of a larger reporting population. The most frequent user types who reported ADRs through the apps were HCPs, such as pharmacists (5/17), medical doctors (4/17) and other HCPs (4/17). Following HCPs, patients or family members were reported as the next most frequent reporter type (3/17). Public health programs were not cited as frequent users in any of the responses.

Quantitative and qualitative impact of app implementation on ADR reporting
The quantitative data of app downloads and ADR reports are summarized in Table 4. The number of downloads tended to be higher for Android users compared to iOS, where both Android and iOS platforms were available. The apps which enabled users to report ADRs to public entities such as NRAs tended to have the higher number of downloads compared to the apps addressing ADRs only to private entities. The average percentage of ADR reports received from the apps compared to reports using all reporting methods varied from 0-60%. This range was broader for LMICs (0-60%) than for HICs (0-5%). A percentage greater than or equal to 5% was reported for eight apps: seven from LMICs and one from HICs. .5% n/a: Not available. a: The same app was named Yellow Card -MHRA in iOS. b: The period for which the data were available for the analysis is indicated. Otherwise, the period from the launch to data extraction was applied. c: Only aggregated data including the number of downloads in Android and iOS were available.
More than half of the respondents said that they had observed a post-launch upward trend in the overall number of ADR reports, not only for those received through the apps but also from other reporting tools, such as paper-based reporting (9/17). Figure 3 shows qualitative differences in "essential elements" appropriately lled in for ADR reports submitted from apps and in paper form. Some 65% of respondents (11/17) said that the number of elements appropriately lled in on reports submitted via app was the same as that for paper reports. Six respondents said that all of the four minimum elements were appropriately lled in on reports submitted using the apps and paper forms.

Real-world experiences with ADR reporting apps
The respondents of the online survey shared the pros and cons of their ADR reporting apps ( Table 5). The inputs from the respondents were synthesized into the following themes: • Apps include functions that facilitate data entry, such as dropdown menus, lists of medicines, and a "save" option to continue reporting later.
• Information can be further reported in detail through attachments.
• Storage of apps requires a small amount of data memory.
• Apps provide adequate space for case narrative compared to paper forms with limited space.
• App collects only limited ADR information and must be complemented by comprehensive reporting by an ADR monitoring centre.
• App takes longer to ll in than paper forms. Di cult to ll in electronic forms on small mobile phone.

Quality of reports
• Apps eliminate quality issues such as missing reporter names and drug names due to mandatory elds.
No comments made.

Accessibility
• Apps make ADR reporting more accessible to all segments of society, such as patients and healthcare professionals, as anyone with a smartphone can report.
• O ine features allow users to use some functions even without an internet connection.
• Translation of apps into local languages makes them more accessible.
• Healthcare professionals must sometimes download the app onto their personal phone, which they may deem inappropriate since the app is work-related.
• App users need internet access to download and use the online functions.
• Users may be reluctant to complete the full user registration process. Also, users may not have email addresses required for the registration.

Innovativeness
• Apps enable us to keep up with the contemporary digital world.
• Apps are mostly suitable for the younger generation and users with an inclination to use technology. Technology-averse or older populations may be reticent about using them.
Data transferability and data sharing • ADR reports are received directly to NRAs in national ADR databases, which contributes to preventing the need for manual data entry, thus saving time. This also eliminates possible transcribing errors.
• Apps are structured in the E2B format required for data processing.
• Apps do not allow reporters to save a copy of a report to refer back to once it has been sent and thus to share the report with multiple internal and external entities. This limits administrative reporting processes, which are routinely followed in some countries.

Two-way communication
• Apps provide users with a range of information, such as safety data on medicines and other related news, in a timely manner.
• Apps engage an audience interested in the safety of medicinal products.
• Immediate acknowledgement messages after report submissions are appreciated by users.
• If apps do not display contact information, users cannot get in touch with regulators or relevant bodies for clari cations or questions concerning the reporting.
• Users may fail to read news if the apps do not have a noti cation feature for the news of interest.
• Only a limited audience is likely to download apps for regulatory purpose.

Costs
• ADR reporting in free of charge for reporters (except the fee for data transfer).
• Cuts the cost of distributing reporting tools to the users.
No comment made.

Data security
No comments made.
• In apps that save reports until they are deleted by users, apps may have no control over patient information being uploaded to the Cloud if the device is backed up there.
• Users may not want to provide details, such as their name, institution and contacts.

Simplicity of use and report quality
Respondents said that the apps simpli ed ADR reporting, making it quick and easy, and reducing delays in reporting to a data addressee such as a competent authority. The simplicity was enhanced by digital features, such as drop-down menus, a de ned drug list, and data le attachment capability. De ning mandatory reporting elds reduced issues of missing data and increased the overall quality of reports. However, the limited information collected by such simple reporting apps needed to be complemented by more comprehensive reporting afterward.
Innovativeness and accessibility of reporting Feedback indicated that apps make ADR reporting accessible to a broader population since anyone with a smartphone can download the app and report ADRs. Although a feature enabled the use of some functions without an internet connection, some internet connectivity was needed later to upload the o ine activity so it could be transmitted. Respondents in countries with limited internet connectivity reported persistent di culties in using the apps.
Respondents said the younger generation and/or users with an inclination to use technology were more likely to use apps compared to more technologyaverse or older people.

Data transferability and data sharing
NRAs valued the direct receipt of ADR reports from apps to their databases since it did not require the manual entry of data, thus saving time and potentially avoiding transcribing errors. Moreover, apps structured in the E2B format were convenient for data processing. A limitation was the inability of the app to save a copy of a report for the reporter to refer back to once it had been sent and to thus be able to share the report with other relevant stakeholders.

Two-way communication
Two-way communication enabled owners and end users to communicate drug safety information and other related news in a timely manner, and strengthened the engagement of end users with an interest in drug safety. When end users received an immediate acknowledgment of their report's submission, it bolstered their commitment by making them feel they were making a valuable contribution to pharmacovigilance activities.

Costs
Respondents expressed only positive comments on cost since the apps were free of charge for reporters and reduced the cost of distributing paper reporting forms.  Table 6. [1] The app and its description were listed on the app store, but it could not be found when we sought to download it.

Discussion
To our knowledge, this is the rst study mapping ADR reporting apps. Of the 22 apps selected for the study, more than 70% were based in LMICs, with the majority in the WHO Africa region.
Although the number of LMICs with pharmacovigilance centers reporting to VigiBase has increased substantially in the last 10 to 15 years, actual reporting per 100,000 population per year is relatively low in LMICs compared to many HICs, which have well-established and more mature pharmacovigilance systems [34][35][36]. Paper-based reporting requires reports to be manually transported by postal services to a pharmacovigilance center once lled out by a reporter, and to be manually entered into a data management system, which is one of the most time-consuming activities in the reporting process [37]. It is likely to result in quantitative and temporal challenges, i.e. delay from the time of the onset of an ADR to the time the report reaches VigiBase [38]. Moreover, LMICs face qualitative challenges due to poorly documented reports [39]. Needless to say, the comprehensiveness of a report plays a critical role in pharmacovigilance in order to aid in the detection of signals [40].

Impact of key reporting features
This study showed that ADR reporting apps offer unique added value compared to conventional reporting tools, such as paper-based formats, through various features, and they were likely to support countries to more e ciently collect data. One of the identi ed features was a standardized format meeting E2B standards [41]. Hence data collected by the apps were easily transferable to the E2B-compatible national data management system and, moreover, the ow of data to VigiBase would be facilitated, thereby addressing the challenges that LMICs face. In fact, according to the online survey results, LMICs were more likely than HICs to take advantage of smartphone-based apps features for reporting. The survey respondents also appreciated that the apps supported data transmission and improved report quality in LMICs. However, not all respondents were able to provide information to thoroughly describe the direct impact of the apps. Further investigation would be warranted on outcomes, such as the time from event onset to data submission on VigiBase, and the report quality appraised through various measurements [39,40,42] between those received from ADR reporting apps and those via conventional routes.

Contribution in strengthening reporting culture
Notably, in this study, more than 50% of the survey respondents commented that apps contributed to an increase in the number of all ADR reports regardless of the type of reporting tool. The upward quantitative trend of all ICSRs in the post-launch periods was also con rmed by the VigiBase search.
As mentioned in the introduction, several countries have been supported by the WHO to roll out the ADR reporting app, Med Safety. The rst country to pilot the app was Burkina Faso in 2017 [43,44], initially as part of the seasonal malaria chemoprevention campaign. Promotional campaigning and trainings were integrated into app roll-out as part of a national malaria disease program. Along with the launch, an extensive campaign was also conducted in Uganda, including a month-long intensive mass media campaign through television, radio, local newspapers and a press conference. The launch attracted hundreds of stakeholders in pharmacovigilance [45]. ADR reporting tools based on user-friendly technology could be appealing to the public and strengthening the culture of reporting. Moreover, integration of pharmacovigilance into the work of other relevant partners, such as public health programs, should have been an indispensable contributor to successful app implementation, and might have indirectly resulted in a rise in the overall number of reports.

Importance of understanding the target environment and users
Although o ine app function was cited as a positive driver in in uencing patients and HCPs to use the ADR reporting app in the Innovative Medicines Initiative (IMI) WEB-Recognizing Adverse Drug Reactions (RADR) project [46,47], this study reported persistent di culties due to internet connectivity issues in some geographical settings. In this survey, this challenge was reported particularly by LMICs and given as one of the reasons for the low use of the app despite the o ine function. Although it was reported that there was a high coverage of fourth generation broadband cellular network technology (4G) in LMICs (82% of the population), it is also known that such services are not affordable in more than half of LMICs [48,49]. In addition, internet stability is a challenge due to unexpected technical issues and sporadic political restrictions [50,51]. App use could also be in uenced by individual socio-demographic status and level of interest [46], and is preferred by the younger generation and those with an interest in the technology. Additional barriers exist in LMICs due to rural-urban differences and gender gaps in mobile internet use, as well as a lack of literacy and digital skills [48]. It may be important to consider the level of digital literacy required to master the settings when introducing an app, to identify additional follow-on support, such as training, that might be required to ensure optimal app implementation and use.

Importance of critically appraising cost-effectiveness
The overall bene t of the apps should be examined alongside the nancial affordability of the technology. Conventional paper reporting incurs various production costs, such as for paper, printing, distribution and postage. In addition to these direct costs, the indirect costs related to the human resources, time and effort required to manually process each paper form received would not be negligible [37]. The cost saving on these items that the apps offer was positively appraised in our online survey. Nevertheless, apps incur their own costs throughout implementation, e.g. pre-development research, design, technical development, testing, deployment, and continual support and maintenance [52]. To ensure that NRAs make cost-effective decisions, we highlight the importance of considering the cost as well as the bene ts of an app when deciding whether to develop and implement it.

Study limitations
Although this study presents globally generalizable evidence based on a robust systematic methodology, the results should be cautiously interpreted due to their limitations. Firstly, although the number of hits from the searches of the Google Play Store and App Store varied depending on the search terms, their seemed to be a ceiling of about 250 items in the displayable hit numbers, especially in the Google Play Store. As a result, it may be possible that some apps did not appear in our searches. Secondly, we ran the searches using English terms, possibly excluding some of the non-English apps, although apps available only in languages other than English were also picked up by our search strategy and included in the study. Also, searches were performed from a single location (Geneva, Switzerland) and the results may differ from another search location. A supplemental study using search terms translated into various languages, such as the o cial United Nations languages, and performed in different locations worldwide would be worthwhile. Lastly, information technology is advancing rapidly. The selected apps will no doubt be updated and the latest versions currently in use might not be identical to the ones reviewed in this study. Also, additional apps may have been developed in the interim. It would be useful to repeat this survey periodically, to capture innovations and new pharmacovigilance apps.

Conclusions
There are various kinds of ADR reporting tools. App-based ADR reporting tools are becoming more popular in different regions of the world and they contribute to ADR reporting through technological features. They also strengthen the overall culture of ADR reporting in their appeal to a wider group of reporters and due to the ease of reporting. ADR reporting apps have the potential to support pharmacovigilance activity; however, in launching an app, it is important to consider features and functions that can contribute to a qualitative and quantitative improvement in reporting, and considering the proposed user group, any training needs, and the costs involved to develop, launch and maintain the app. Moreover, further post-implementation studies would help assess the long-term impact of app-based tools and how these can be sustained.

Disclaimer
The views presented in the manuscript do not necessarily represent the decisions or policies of the World Health Organization.

Figure 3
Comparison of essential elements* appropriately lled in for ADR reports on apps and in paper form