ISO/IEC 25010- based Quality evaluation of three mobile applications for reproductive health services in Morocco

DOI: https://doi.org/10.21203/rs.3.rs-2720323/v1

Abstract

Background and objective. Mobile apps for postnatal, contraception, and prenatal care are crucial for enhancing the health of mothers and babies. These apps provide daily monitoring and resources, including reliable infant care, feeding, and maternal recovery information. Contraception apps give individuals the means to make informed family planning decisions and access various contraception methods. Prenatal apps offer expectant mothers educational resources, tracking tools for pregnancy milestones, and a connection to healthcare providers for important check-ups and screenings, all while boosting doctor-patient communication. The apps also often include discussion forums where users can connect and share experiences. By providing access to care, information, and support, these mobile apps significantly ensure mothers and babies receive the necessary resources and care to thrive. This study assesses the software quality (SQ) of three reproductive health monitoring mobile apps using ISO/IEC 25010 and ISO/IEC 25023 standards. The aim of the study is to identify the strengths and weaknesses of each app in terms of SQ, with the goal of providing valuable information for users who are considering using one of these apps. By examining the positive aspects of each app, the study provides insights into what makes them a valuable choice. Similarly, by identifying negative points in terms of quality, the study provides an opportunity to identify areas for improvement, so that the apps can be evaluated and improved over time.

Methods. A three-step analysis process was implemented, of which the initial step involved examining the sub-features and features of the ISO/IEC 25010 standard for Software Product Quality (SPQ). The next step involved developing a checklist that summarizes the effect of the three mobile apps on SPQ. Finally, a calculation was performed to assess the impact of each requirement block on both the SPQ and the Quality in Use model.

Results. The findings indicate the Functional Suitability SPQ characteristic is greatly influenced by the predefined requirements, followed by the Operability, Performance Efficiency, Reliability, and Security SQ characteristics. Therefore, developers, designers, and testers must factor in these requirements throughout the development cycle of health mobile apps to produce a high-quality, patient-centered product with significant added value.

Conclusion. This study generated various requirements for pregnancy monitoring and childcare apps, providing a useful reference for developers in creating high-quality apps also assessing the quality of these kinds of apps. The software development process must consider SPQ characteristics to create a high-quality product with crucial KPIs, leading to better evaluations in app stores and increased user adoption and satisfaction.

1. Introduction

Properly monitoring and managing a pregnancy leads to a successful birth and the good health of both the mother and child. This can be accomplished by mothers, midwives, and healthcare professionals for maintaining good health during pregnancy by preventing complications and providing essential Reproductive Health Services. Access to high-quality reproductive health services plays a crucial role in maintaining an individual's overall well-being, particularly for those people who are unable to attend medical appointments due to their busy schedules or distance [1]. Moreover, according to several studies, the use of technology and eHealth apps increases the quality of healthcare services and facilitates the integration of data for monitoring and assessing the patient's health status [2]–[4]. Electronic Health Records (EHRs), for example, play a significant role in improving clinical care, preventing complications, and promoting quality improvement. Furthermore, implementing EHRs can help bridge the quality gap in the healthcare systems of low- and middle-income countries (LMICs)[5]. The importance of eHealth has been underscored by the COVID-19 pandemic, which has led to a surge in the use of telemedicine through mobile health (mHealth) applications. The successful implementation of these technologies has demonstrated their efficacy in managing public health during this crisis [6]–[8].

Health monitoring apps for pregnant women improve their lives by enhancing communication with doctors, simplifying the monitoring process, and allowing easy tracking of information for quick medical intervention in emergencies. Out of 73% of parents prefer to receive electronic discharge advice and digital technology for research, making digital technology the preferred choice for research among the majority of parents[9]. Especially that the use of smartphone apps has seen rapid growth, with 6.6 billion users in 2022 (4.9% annual growth and 79% increase from 2.9 billion users in 2016), and is expected to continue to rise in the coming years [10].

In 2021, there are over 400,000 health-related mobile apps accessible in the primary app stores, with 25% of them being associated with health problems, notably pregnancy mobile apps. According to a systematic review conducted in Australia, there were 2052 pregnancy-related mobile apps available on platforms in Australia in 2017. In line with other studies showing that women often turn to digital resources, such as mobile apps, for monitoring during pregnancy and for tracking menstrual cycles [11]. With the Internet's emergence and growth of the new web, and mobile technologies, access to maternity information has become very easy and available [12]. In addition, studies have shown that m-Health apps can have a positive impact on both healthy lifestyles and medical care. Lifestyle m-Health apps have been found to be well-accepted and feasible, and have even had positive effects on health behaviors [13].

In our previous studies[14]–[16] and research as part of the PEER 7-246 project [17], three Android apps were developed to improve and facilitate access to reproductive health services for women in Morocco. The goal is to enable women to take control of their reproductive health regardless of their circumstances and conditions. Each app provides a specialized service (The different screens of the mobile apps can be found in the Appendix in Figure 13, Figure 14 and Figure 15):

The primary objective of this study is to evaluate the SQ of the three mobile apps "MyContraception", "MaGrossesse", and "Mamma&Baby" using the ISO/IEC 25010 software quality standard. The paper focuses on the impact of requirements on the SQ of these apps. To do this, four research questions (RQs) have been defined to analyze the SQ of the three apps as presented in Table 1. By exploiting common requirements, developers are able to create mobile apps that support and simplify the pregnancy period for women, testers also use these requirements as a basis to ensure the app meets quality standards and provides accurate information to users.

 Table1: Research Questions (RQs)

Research Question (RQ)

Motivation

RQ1.What is the impact of pregnancy monitoring, newborn baby care, and contraception mobile apps requirements on the SPQ?

RQ2. What are the requirements that have a significant impact on SPQ?

RQ3. What are the other features that should be added to the three apps?

RQ4. What are the common features among the three apps?

To examine the effect of mobile apps for pregnancy monitoring, newborn care, and contraception on the SPQ.

To determine the key features of the three apps that have the greatest effect on the SPQ.

What areas need improvement in order to achieve a high-quality product.

Emphasize the common features in mobile apps of the same type.

 

To sum up, the study was conducted according to the following stages: (1) Extraction of the characteristics and sub-characteristics of the Software Product Quality (SPQ) model and Quality in Use model; (2) Description of requirements for mobile apps related to contraception, prenatal, and postnatal care; (3) An analysis process consisting of three steps, which includes examining the ISO/IEC 25010 standard for SPQ, creating a checklist of requirements for the SPQ model of ISO/IEC 25010 for mobile apps related to pregnancy monitoring, newborn baby care, and contraception, and determining the degree to which the requirements affect the SPQ; and (4) Discussion of the main results and findings of the study.

The remainder of this article is organized as follows: Section 2 gives an overview of the SQ standards employed. Section 3 outlines the requirements for mobile apps related to contraception, prenatal, and postnatal care. The impact analysis process for pregnancy monitoring, childcare, and contraception apps on SQ is explained in detail in Section 4. The findings of this impact analysis are summarized in Section 5. The discussion of these results is presented in Section 6. Section 7 includes research and practical implications, as well as limitations of the study. Finally, Section 8 provides the conclusions and suggests future work.

2. Standards Used

2.1 ISO/IEC 25010 Quality Model

The SQuaRE (Software Product Quality Requirements and Evaluation) family of international standards includes the ISO/IEC 25010, which was introduced in 2011 to replace the ISO/IEC 9126. Although it was technically updated and republished in 2001, ISO/IEC 9126 is an international standard for the specification and evaluation of SPQ [19, p. 25010].

The standard specifies two models: the Software Product Quality (SPQ) model and the Quality In Use model. The SPQ model focuses on the quality of the product and includes eight characteristics [12]. Figure 1 presents the characteristics of the SPQ model, with their corresponding sub-characteristics displayed in Fig. 3. On the other hand, the Quality in Use model is related to the quality of the product when used by its intended users and composed of five characteristics [11]. Figure 2 illustrates the characteristics of the Quality in Use model, while Fig. 4 illustrates its sub-characteristics.

Some characteristics are subdivided into sub-characteristics related to the outcome of using the product within a particular context of use [11].

ISO 25010: A valuable framework for enterprise software teams to design and deliver successful software solutions. This standard divides SQ into various sub-aspects, allowing developers to establish and design tailored metrics and measurements to accurately assess both functional and non-functional requirements [20].

2.2 ISO/IEC 25023: Assessment of System and SPQ

ISO/IEC 25023:2016 is a SQuaRE standard that replaces ISO 9126-2 and ISO 9126-3. When combined with ISO/IEC 25010, ISO/IEC 25023 offers a collection of metrics for quantitatively measuring the quality of systems and software products, based on the characteristics and sub-characteristics defined in ISO/IEC 25010[21].

3. Requirements Of Contraception, Prenatal And Postnatal Mobile Apps

This section summarizes crucial requirements for pregnancy monitoring and child care apps. These requirements are based on the results of various research papers that have been published before. In the Research [15] a set of common features for mobile personal health records (mPHR) has been identified to improve contraceptive fit, with the goal of defining a set of components for a specific contraception mPHR [15]. More features were extracted from the study [15] in their review and analysis, they evaluated the functionalities and technical characteristics of mobile health apps for postnatal care [22]. The outcome of the studies outlined the functional and non-functional requirements of a pregnancy monitoring app [23] [24].

 The main requirements for pregnancy monitoring and child care mobile apps have been grouped into eight categories as shown in Figure 5, each with its own set of sub-requirements. The primary objective of these requirements is to establish a checklist to evaluate how each predetermined requirement impacts the external sub-characteristics of the product quality model, as illustrated in Table 2 in the appendix.

"Mobile app accessibility" refers to the effort to make mobile apps more accessible to all users, including those with disabilities. It measures the ability of individuals with varying abilities and features to use a product or system to attain a predefined objective within a particular context [25].

 The following are the requirements for the app’s accessibility(AA): 

                          B.       User’s Personal Information

To effectively use a pregnancy monitoring mobile app, users must first create a personal account, which contains their personal information (PD): PD1. The user's full name, PD2. The user's date of birth, PD3. The user's email address, PD4. The user's phone number.

                         C.        Health Information of the User

The user's health record (HI) should contain the following medical information: HI1. illness history, HI2. blood group, HI3. psychological disorders, HI4. emotional disorders, HI5. type of delivery, HI6. number of childbirths, HI7. family illness history, HI8. vaccines, and HI9. contraceptive method.

                         D.       Baby’s Personal Information

For the postnatal mobile app, it must contain the following personal information of the newborn (NI): NI1. full name, NI2. gender, NI3. date of birth.

                         E.       Baby’s Physical Body Quantitative Data

The baby's physical body quantitative data (ND) is very crucial to have total monitoring and tracking of the baby's growth. Some of this data includes ND1. weight, ND2. height, and ND3. head circumference. 

                         F.        Baby’s Health Information

The baby's health record is supposed to contain this health information (NH): NH1.   sleep duration, NH2.vaccines, NH3. blood group.

                         G.       The User's Action

The requirements document should include a set of inherent behaviors referred to as "user's actions" (UA). These actions represent the various interactions that users can potentially undertake with the mobile app, regardless of whether they are actually able to perform them or not, and they should be explicitly outlined in the document: UA1. use login credentials to authenticate, UA2. creating information, UA3. Updating and modifying information, UA4. deleting information, UA5. uploading image, UA6. sharing information, UA7. backup data.

                         H.       App's features

This requirement group includes prenatal, postnatal, and contraception mobile app features.                        

                             i.   Contraception mobile app features

This sub-block identifies the functionalities of the "my contraception" application. It includes: 

                        ii.   Prenatal mobile app features

This sub-block identifies the functionalities of the "MaGrossessemobile app. It contains:

                        iii.   Postnatal mobile app features

This sub-block defines the features of the "Mamma&Baby" mobile application which include: 

4. Impact Analysis Of Pregnancy Monitoring, Child Care, And Contraception Mobile Applications

This section describes the methodology used to assess the impact of requirements on the SPQ of three mobile apps. The approach follows the methodology presented in a previous study [17],[26], which applied the ISO/IEC 25010 standard to Mobile Personal Health Records (mPHR). The objective of this analysis is to address the RQs by providing sample calculations.

Step1. The analysis of the sub-characteristics and characteristics of the ISO/IEC 25010 standard for SPQ

To evaluate the SPQ, the ISO/IEC 25010 standard was utilized, requiring an in-depth analysis of the external SQ characteristic and sub-characteristic definitions. The ISO/IEC 25023 was also employed in conjunction with ISO/IEC 25010 to provide a quantitative assessment of the SQ in terms of its characteristics and sub-characteristics [12].

Step2. Checklist of ISO/IEC 25010 SPQ model requirements for pregnancy monitoring, newborn baby care, and contraception mobile apps

A checklist has been developed (as demonstrated in Table 2 in Appendix) to evaluate how each pre-determined requirement affects the external sub-characteristics of the product quality model. If the variables used to calculate the external metric are impacted by at least one requirement, it is concluded that the SPQ sub-characteristic is impacted by this requirement.

Step 3. Calculation of the degree of impact of pregnancy monitoring, newborn baby care, and contraception mobile apps’ requirements on SPQ

Calculated impact degrees fall into three categories:

i. "ID(EC,B)= ∑ID(EC,B)/N(R)", in which:

ii. "ID(EC, R) = N(EsC, R) / N(EsC)", where

iii. "ID(EsC, B) = ∑ID(EsC,R) / N(R)", where :

The values of N(R) and N(EsC) are determined by using the checklist we created in step 2 (Table 2 in Appendix), following this analysis, each impact degree is divided into five different groups as illustrated in Table 3

Table 3

Impact Degrees and Classification Group

Impact Degree

Classification Group

[0.9,1]

Very high

[0.7,0.89]

High

[0.4,0.69]

Moderate

[0.2,0.39]

Low

[0,0.19]

Very low

The following example has been provided, which focuses on calculating the degree to which the POF block of requirements has an impact, nineteen functional suitability (FS) requirements are contained in this block, to demonstrate the analysis procedure outlined above.

Initially, the impact level of each requirement within the POF block on the FS has been determined using this equation: ID(FS, POF) = \(\sum _{{i}=1}^{{i}=19}\left(\mathbf{I}\mathbf{D}\right(\mathbf{F}\mathbf{S}, \mathbf{P}\mathbf{O}\mathbf{F}\mathbf{i}\left)\right) /19\).

According to Table 2 in the Appendix, the requirements POF6 to POF17 influence only the Appropriateness sub-characteristic, with impact degrees of:

ID (FS, POF6) = ID (FS, POF7) = ID (FS, POF9) = ID (FS, POF10) = ID (FS, POF12) = ID (FS, POF13) = ID (FS, POF14) = ID (FS, POF16) = ID (FS, POF17)= ½.

The remainder of the requirements has an impact on both Appropriateness and Accuracy sub-characteristics with the following values of impact degrees:

ID (FS, POF1) = ID (FS, POF2) = ID (FS, POF3) = ID (FS, POF4) = ID (FS, POF5) = ID (FS, POF8) = ID (FS, POF11) = ID (FS, POF15) = ID (FS, POF18) = ID (FS, POF19) = 2/2 = 1.

The degree of impact of the whole block of requirement POF on Functional Suitability is, therefore:

ID (FS, POF)=(1 + 1 + 1 + 1 + 1 + 0.5 + 0.5 + 1 + 0.5 + 0.5 + 1 + 0.5 + 0.5 + 0.5 + 1 + 0.5 + 0.5 + 1 + 1)/19 = 0.76.

On the other hand, the POF block impact degree on the Accuracy sub-characteristic (Ac) has been calculated using the following equation: ID(Ac, POF)=\(\sum _{{i}=1}^{{i}=19}\mathbf{I}\mathbf{D}(\mathbf{A}\mathbf{c}, \mathbf{P}\mathbf{O}\mathbf{F}\mathbf{i})/19\).

According to the checklist, all the requirements impact the Appropriateness sub-characteristic (Ap), hence,

ID (Ap, POF1) = ID(Ap, POF2) = ID(Ap, POF3) = ID(Ap, POF4) = ID(Ap, POF5) = ID(Ap, POF6) = ID(Ap, POF7) = ID(Ap,POF8) = ID(Ap,POF9) = ID(Ap,POF10) = ID(Ap, POF11) = ID(Ap,POF12) = ID(Ap, POF13) = ID(Ap, POF14) = ID(Ap, POF15) = ID(Ap, POF16) = ID(Ap, POF17) = ID(Ap, POF18) = ID(Ap, POF19) = 1.

Therefore, ID (Ac, POF) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)/19 = 1.

Based on the classification of the degrees of impact shown the Table 3, we can see that the impact of the block of requirements POF on Functional Suitability DI (FS, POF) is high while the impact degree of this block on the Accuracy sub-characteristic DI (Ac, POF) is very high.

5. Results

This section presents the outcomes of the impact analysis process conducted on pregnancy monitoring, postnatal care, and contraception mobile apps requirements, with the purpose of addressing the primary RQs stated in the introduction of this study. The analysis examines the effect of these requirements on the SPQ characteristics.

The checklist shown in Table 2 in the appendix contains 66 requirements for the three reproductive healthcare areas (pregnancy monitoring, postnatal, and contraception) and a total of 30 external quality sub-characteristics. 

As shown in Figure 6, the impact of each requirement of the "MyContraception" mobile app on the external quality characteristics is depicted. It is noteworthy that among all the SQ characteristics, only Functional Suitability is affected by all the requirements.  95% of the requirements affect the Operability characteristic, whereas the Performance Efficiency and Security SQ characteristics are impacted by 87%, and Reliability by 85%.  Whereas 74% of requirements affect Maintainability, while Transferability and Compatibility are impacted by only 10% and 8% of requirements, respectively. SQ is largely affected by the requirements UA1, followed by PD1, PD2, PD3, PD4, and CF5. However, the requirement that has a small impact on the SQ characteristics is AA3.

Figure 7 presents the impact of each requirement on the external characteristics of the "Mamma&Baby" postnatal care mobile app. It is important to mention that only Functional Suitability among all the external characteristics is impacted by all the requirements. 97% of the requirements have an impact on Operability. Performance Efficiency, Reliability, and Maintainability are influenced by 92%, 90%, and 84% of the requirements, respectively. Security is influenced by 65% of the requirements, and Compatibility by 35% while the lowest degree of impact (6%) is obtained by Transferability. UA1 is the requirement that significantly affects SPQ, followed by POF19. On the other hand, AA3 has the least impact on the SQ characteristics.

Figure 8 illustrates how each requirement affects the external characteristics of the prenatal care mobile app "MaGrossesse". All requirements have an impact on the Functional Suitability SQ characteristic, with 95% of them having an impact on Operability. Performance Efficiency and Reliability are both impacted by 88% and 85% of the requirements, respectively, followed by Security (76%) and Maintainability (75%), while Compatibility is influenced by only 17% of the requirements, and the lowest impacted characteristic is Transferability with a degree of 10%. SPQ is mostly impacted by the requirement UA1, followed by PF5, while AA3 has the weakest impact on external quality characteristics.

Figure 9 displays the effect of the requirements for pregnancy monitoring, postnatal care, and contraception mobile apps on the SPQ characteristics. The results show that Functional Suitability is the most affected characteristic by the requirements, followed by Operability, Performance Efficiency and Reliability. On the other hand, Transferability is the characteristic that is least influenced by the requirements.

Based on Figure 10, all requirement blocks affect the Appropriateness, Ease of Use, Appropriateness recognizability, and Technical Accessibility sub-characteristics. Whereas the three sub-characteristics of Transferability, namely Portability, Adaptability, and Installability are affected only by the block of requirements AA, none of the other requirements blocks affect these three sub-characteristics. Moreover, as a security sub-characteristic confidentiality is significantly impacted. The influence of PF and UA blocks on Interoperability sub-characteristic is equal. Regarding Operability characteristic, Ease of use, technical accessibility, and Appropriateness recognizability are its three sub-characteristics that are highly impacted by all the requirements blocks. All the requirement blocks, excluding AA, influence the sub-characteristics of Reliability. 

The PD block has the strongest impact on the SQ as shown in Figure 11. Out of the eight quality characteristics, Security and Transferability are the ones that are least impacted by the requirement blocks because of their low and extremely low degrees. All requirements, except AA, have a significant influence on Reliability, whereas only block AA has an impact on Transferability.

Figure 12 shows how the requirements of the three apps influence the 30 external quality sub-characteristics. As shown in this figure, appropriateness is the most impacted sub-characteristic followed by Fault tolerance, Time behavior, Availability, Ease of use, Recoverability, Appropriateness recognizability, and Resource utilization. However, the sub-characteristics of Learnability, Helpfulness, Integrity, Coexistence, Interoperability, Reusability, Analyzability, Portability, Adaptability, and Installability are least affected by the requirements.

6. Discussion

The findings of the study, which focused on the requirements of three reproductive healthcare mobile apps "MaGrossesse", "MyContraception", and "Mamma&Baby" are discussed in this section. The requirements were extracted, refined, and organized based on relevant studies in the three reproductive healthcare domains.

The SQ of the three apps was evaluated by assessing the impact of the specified requirements on eight quality characteristics and their related sub- characteristics, as defined in the ISO/IEC 25010 model and ISO/IEC 25023 standard. As presented above in section 3, three categories of requirements have been identified to define those related to each domain of the three reproductive healthcare: Prenatal, Contraception, and Postnatal. To determine the extent to which each requirement affects the external quality attributes, the analysis process outlined in section 4 has been applied to all requirements, which are categorized as SQ requirements, except the OS version (AA2) and cost (AA3). The data analysis and the generation of graphics have been based on the use of Statistical Package for the Social Sciences software version 22.2 (SPSS Statistics 22.0)

RQ1.What is the impact of pregnancy monitoring, newborn baby care, and contraception mobile apps requirements on the SPQ?

After completing the application of the impact analysis process, it was discovered that the blocks PD, PF, and POF significantly influenced the SQ characteristics. There has been a significant rise in the number of mobile apps related to health, offering advanced tools and services for decision support healthcare systems based on the user's situation, particularly in the medical domain. These apps operate using both implicit and explicit data collected from the user and their environment, and they store confidential information about the patient [27]. This implies the importance of implementing security ([28],[29]) and privacy as non-functional requirements and considering it during the development of the mobile app, as failure impacts negatively stakeholders and users alike.

In addition to data security and privacy, eHealth and m-health applications must adhere to strict regulations and standards, such as HIPAA (Health Insurance Portability and Accountability Act) in the United States, to ensure that patient data is handled appropriately and within legal boundaries. SQ is necessary to ensure that the mobile app meets these requirements and complies with the relevant regulations and standards [30], [31]. Therefore, it is essential to prioritize SQ characteristics such as security, privacy, and portability in the development of mobile health apps. By doing so, developers can ensure that their apps meet the highest standards of quality and that users can trust them with their sensitive information which enhances the popularity of the mobile app and increase the number of downloads and positive reviews [25].

RQ2. What are the requirements that have a significant impact on SPQ?

Regarding the specifications considered as app features, an integrated social network for moms to help each other, and share experiences, and problems (POF19) for the postnatal app is the bloc POF requirement that has the highest impact on SPQ (Fig. 7, Fig. 10, and Fig. 11). Research indicates that individuals who use social media benefit from health recommendation services for minor issues, particularly busy individuals who prefer it to in-person consultations which can lead to misdiagnoses ([32],[33]). A mobile discussion forum is one of the most interesting features, especially in pregnancy monitoring and childcare apps ([34],[35]). This in-app open discussion area is an opportunity for pregnant women and new mothers to ask questions, discuss their potential fears and concerns about their pregnancy or postnatal period, and share their experiences to receive real-time guidance in times of doubt.

Functional Suitability, the most impacted characteristic by the requirements, ensures that apps meet user requirements, provide expected results, and offer features such as ease of use and reliability. Numerous studies have been carried out to identify an effective approach for evaluating ISO/IEC 25010's critical characteristic of functional suitability and usability ([26],[36]). The findings of these studies have led to the development of a comprehensive evaluation methodology, procedure, and a set of tools that enable a thorough assessment of functional suitability ([37]–[39]).

During the design and development stages, Operability emerges as the second most affected characteristic by the requirements in the three apps, based on the observations made in Fig. 9. Its significance lies in its critical role in improving user engagement, satisfaction, health outcomes, and app adoption while reducing errors. The ease of use of health apps is crucial for users to achieve their goals, as pointed out by studies [38], [39], and [40]. Furthermore, Fig. 9 highlights the considerable influence of requirements on Performance Efficiency and Reliability, which are essential for ensuring a positive user experience, fostering trust, driving revenue, and maintaining competitiveness in the mobile app market. A study conducted on 115 organizations in Italy confirmed that during crisis conditions, IT performance is significantly influenced by usage reliability and support service reliability [41].

Efficient performance also has a significant as shown in Fig. 9, and 11, it helps minimizing battery usage, process sensitive data quickly and securely, and providing timely and accurate information to users. There are several factors that can contribute to the efficient performance of a mobile app, including optimizing code by respecting the best practices of programming ( comment the code, avoid duplications, using design patterns, etc) minimizing network requests, Efficient memory usage, Battery optimization, Testing and debugging[40]–[42]. Efficient performance of a mobile app is determined by its speed, responsiveness, and resource utilization, all of which are essential for a positive user experience.

RQ3. What are the other features that should be added to the three apps?

According to Figs. 6, 7, 8, and 9, transferability and compatibility are the characteristics that appear to be the least affected by requirements. Ensuring transferability and compatibility helps to ensure that the app is accessible, functional, and able to provide positive user experiences.

To ensure transferability for health apps, developers can use standard data formats that are compatible with multiple platforms, such as the Health Level Seven International (HL7) standard for electronic health records (EHRs)[43]–[46]. They can also use cloud-based storage solutions to allow users to access their data from any device with an internet connection[47]–[49].

To ensure compatibility, developers can use standard coding practices and programming languages that are compatible with multiple platforms like JavaScript, Dart, Xamarin, Kotlin and others [50]–[52]. They can also test the app on various devices and operating systems to ensure that it works properly on all of them[53]–[56].In addition, developers can use responsive design techniques to ensure that the app's user interface and features adapt to different screen sizes and resolutions.

This passage is referring to a reform of the authentication mechanism of a system. The authentication (UA4) was found to have significant security flaws, and the impact of these flaws on the system is significant (as shown in Figs. 7, 10, and 11). To address these flaws, Additional security steps are being suggested to offer enhanced security for user credentials. Some options include utilizing a secure question, smart card technology, or biometric verification. These alternatives provide various benefits and are aimed at increasing the protection of the user's information and reducing the possibility of unauthorized access, thereby safeguarding confidential data [57].

Apps should not only provide useful features, but also secure their users' data. A block-chain and Distributed Ledger-based Improved Biomedical Security system (BDL-IBS), has been suggested in study [58] as a way to increase privacy and security in healthcare. The research found that using block-chain technology offers fast, easy, and safe data exchange, improving privacy and security for patients.

RQ4. What are the common features among the three apps?

Keeping track of your medical history might be stressful. Keeping track of all your doctor's appointments, conversations, tests, lab results, and other treatments may sound like a lot of work, especially for people managing complicated or various health conditions ([59],[60]), which implies the importance of this feature in mobile apps ([61],[62]). In the three apps "Mamma&Baby", "MaGrossesse", and "Mycontraception" there is a common feature which is the capability of tracking your medical history (CF5) for the contraception app, (PF1) for the prenatal app, and (POF6) for the postnatal app, can be added as a crucial source of security shortcomings beside the authentication.

Reminders have a crucial function in mobile applications by assisting users in remembering vital tasks and events, avoiding forgetfulness, enhancing the user experience, boosting engagement, and improving the reliability of the application. The apps "Mycontraception" and "Mamma&Baby" share a common feature (CF3) and (POF5), which is the reminder feature that enables women to create alerts or notifications for essential tasks, such as taking medication, attending appointments, staying on track with their breastfeeding goals, or meeting deadlines. Reminders, including mobile phone-based interventions and patient-centered reminders, have been shown to be effective in improving medication adherence and follow-up care in postnatal contraception and prenatal care settings[63]–[65].

7. Research And Practice Implications

This study generated several requirements for pregnancy monitoring and childcare apps that may be exploited by developers and SQ evaluators for several needs. The process of software development might be followed by an evaluation of the SPQ to deliver a product with a high level of quality, more stable, and bug-free version. The SQ requirements specification should provide quantitative descriptions of the monitoring and childcare requirements for external SQ characteristics using external measurements. These descriptions should subsequently be used as criteria for evaluating the requirements. The checklist used in this study to evaluate how pregnancy monitoring and childcare requirements affect SQ characteristics and SQ sub-characteristics may be very helpful to developers of these kinds of mobile apps, as well as testers of web or mobile software products: unit testing, integration testing, functional testing, etc. Moreover, the concepts discussed and standards used may be taken into account by stakeholders to develop the essential guidelines that help to improve the overall standard of pregnancy monitoring and childcare apps.

8. Limitations Of The Study

This evaluation of the effect of the requirements for three mobile apps on the SPQ has been based on the initial versions of the apps, which included bugs and some crashes. This is the first limitation of the study. Computer science students conducted several testing sessions to evaluate the apps and fill out app-related questionnaires containing questions about the current functionalities, bug lists, areas for improvement, problems, and an open space for general comments. Based on the results of the testing sessions, the SPM team has begun making improvements to the apps, which involves corrective and evolutionary maintenance and testing to achieve a stable and robust version of the apps. Other unmentioned requirements may also have a significant impact on the research, which constitutes another limitation. Although the limitations of this study might have had a minor impact on the results, it is still believed that the study's findings could be valuable for future endeavors.

9. Conclusion And Future Work

In this study, the three apps "Mamma&Baby", "MaGrossesse", and "Mycontraception" were introduced and used to identify 66 requirements for pregnancy monitoring, contraception, and postnatal care. The degree of impact of these requirements was evaluated using the ISO/IEC 25010 and ISO/IEC 25023 standards, which are used to measure software product quality (SPQ). A checklist was developed, which included the requirements and the eight SQ characteristics and sub-characteristics specified in the ISO/IEC 25010 standard. The findings revealed that specific sub-characteristics such as Transferability, Compatibility, and Maintainability were not considerably impacted by the pregnancy monitoring and childcare requirements. Nonetheless, these requirements had the most significant influence on the Functional Suitability characteristic, followed by Operability, Performance Efficiency, Reliability, and Security.

This study enabled an exploration of the four research questions outlined in Table 1, with the following results:

  1. For RQ1, it was concluded that privacy and security requirements have a significant impact on SPQ.
  2. RQ2, it was determined that the Functional Suitability, Operability, and Efficient Performance are the characteristics most affected by these requirements for the three apps.
  3. RQ3, the need for improving the system's authentication mechanism for enhanced privacy and security was highlighted. It was suggested that native solutions should be used to enforce compatibility and transferability of the apps.
  4. Regarding RQ4, it was found that features such as medical history tracking and reminders were present in all three apps, indicating their importance in healthcare decision-making.

This research highlights the essential characteristics that developers should consider when creating mobile apps related to pregnancy monitoring, childbirth, and contraception. It is suggested that taking into account Security, Privacy, and aspects like Functionality, Operability, and Performance will create a superior product.

The upcoming work focuses on the assessment of the Quality in Use of the three mobile apps designed for pregnancy monitoring and childcare, as described in Section 2 of the ISO/IEC 25010 standard. In order to eliminate technical problems and to facilitate a comparison of the evaluation results on both iOS and Android platforms, we will utilize the most recent versions of these mobile apps. By doing so, we aim to provide a comprehensive and accurate evaluation of the quality of use of these apps. This will enable us to identify any potential areas for improvement and to make recommendations for enhancing the user experience. Additionally, it will provide valuable insights into the effectiveness and efficiency of these apps in supporting users during pregnancy, contraception, and postnatal care.

Declarations

Competing interests 

All authors have declared that no competing interests exist. 

Funding 

The authors received no specific funding for this work.

Acknowledgments 

This work was conducted within the research project PEER 7-246 supported by the US Agency for International Development. The authors would like to thank the National Academy of Science, Engineering, and Medicine, and USAID for their support. 

Author contributions 

All authors contributed to the study conception and design. 

Material preparation, data collection and analysis were performed by [Karima MOUMANE] and [Ali IDRI]. The first draft of the manuscript was written by [Karima MOUMANE] and all authors commented on previous versions of the manuscript. 

All authors read and approved the final manuscript.

References

  1. M. Amiri, I. M. El-Mowafi, T. Chahien, H. Yousef, and L. H. Kobeissi, ‘An overview of the sexual and reproductive health status and service delivery among Syrian refugees in Jordan, nine years since the crisis: a systematic literature review’, Reproductive Health, vol. 17, no. 1, p. 166, Oct. 2020, doi: 10.1186/s12978-020-01005-7.
  2. L. Lazuardi, G. Y. Sanjaya, P. B. Ali, R. G. M. Siahaan, L. Achmad, and H. Wulandari, ‘Interoperability of Health Digitalization: Case Study on Use of Information Technology for Maternal and Child Health Services in Indonesia’, Business Information Systems, pp. 317–327, Jul. 2021, doi: 10.52825/bis.v1i.53.
  3. F. Zsarnoczky-Dulhazi et al., ‘Promising European research results to improve hospitality in healthcare by eHealth’, International Journal of Hospitality Management, p. 103411, Dec. 2022, doi: 10.1016/j.ijhm.2022.103411.
  4. O. Potluka et al., ‘Evaluation of eHealth assistance in-hospital care for improved quality of life in patients’, Evaluation and Program Planning, p. 102261, Feb. 2023, doi: 10.1016/j.evalprogplan.2023.102261.
  5. ‘Implementation of Electronic Health Records During Global Outreach: A Necessary Next Step in Measuring and Improving Quality of Care - ScienceDirect’. https://www.sciencedirect.com/science/article/abs/pii/S0363502321006110 (accessed Feb. 25, 2023).
  6. J. Cao, G. Zhang, and D. Liu, ‘The Impact of Using mHealth Apps on Improving Public Health Satisfaction during the COVID-19 Pandemic: A Digital Content Value Chain Perspective’, Healthcare, vol. 10, no. 3, Art. no. 3, Mar. 2022, doi: 10.3390/healthcare10030479.
  7. A. R. Taha et al., ‘The integration of mHealth technologies in telemedicine during the COVID-19 era: A cross-sectional study’, PLOS ONE, vol. 17, no. 2, p. e0264436, Feb. 2022, doi: 10.1371/journal.pone.0264436.
  8. T. H. Tebeje and J. Klein, ‘Applications of e-Health to Support Person-Centered Health Care at the Time of COVID-19 Pandemic’, Telemedicine and e-Health, vol. 27, no. 2, pp. 150–158, Feb. 2021, doi: 10.1089/tmj.2020.0201.
  9. S. Doyle et al., ‘Efficacy of Digital Health Tools for a Pediatric Patient Registry: Semistructured Interviews and Interface Usability Testing With Parents and Clinicians’, JMIR Formative Research, vol. 6, no. 1, p. e29889, Jan. 2022, doi: 10.2196/29889.
  10. ‘How Many People Have Smartphones? [Apr 2022 Update] | Oberlo’. https://www.oberlo.com/statistics/how-many-people-have-smartphones (accessed Dec. 07, 2022).
  11. L. M. Musgrave, N. V. Kizirian, C. S. E. Homer, and A. Gordon, ‘Mobile Phone Apps in Australia for Improving Pregnancy Outcomes: Systematic Search on App Stores’, JMIR mHealth and uHealth, vol. 8, no. 11, p. e22340, Nov. 2020, doi: 10.2196/22340.
  12. L. Buchanan, E. Anderson, H. X. MBiostat, P. Phongsavan, C. Rissel, and L. M. Wen, ‘Sources of information and the use of mobile applications for health and parenting information during pregnancy: Implications for health promotion’, Health Informatics J, vol. 27, no. 3, p. 14604582211043146, 2021, doi: 10.1177/14604582211043146.
  13. ‘JMIR mHealth and uHealth - The Usability and Effectiveness of Mobile Health Technology–Based Lifestyle and Medical Intervention Apps Supporting Health Care During Pregnancy: Systematic Review’. https://mhealth.jmir.org/2018/4/e109/ (accessed Feb. 04, 2023).
  14. M. Bachiri, A. Idri, T. Rachad, H. Alami, and L. M. Redman, ‘A Complete Prenatal Solution for a Reproductive Health Unit in Morocco’, in Trends and Innovations in Information Systems and Technologies, Cham, 2020, pp. 36–43. doi: 10.1007/978-3-030-45697-9_4.
  15. M. Kharbouch, A. Idri, T. Rachad, H. Alami, L. Redman, and Y. Stelate, ‘MyContraception: An Evidence-Based Contraception mPHR for Better Contraceptive Fit’, in Trends and Innovations in Information Systems and Technologies, Cham, 2020, pp. 86–94. doi: 10.1007/978-3-030-45697-9_9.
  16. L. Sardi, A. Idri, T. Rachad, L. Redman, and H. Alami, ‘Gamified e-Health Solution to Promote Postnatal Care in Morocco’, in Computational Science and Its Applications – ICCSA 2020, Cham, 2020, pp. 931–946. doi: 10.1007/978-3-030-58811-3_66.
  17. ‘Project 7-246 Morocco’. https://sites.nationalacademies.org/PGA/PEER/PEERscience/PGA_189096 (accessed Feb. 25, 2023).
  18. M. Kharbouch, A. Idri, T. Rachad, H. Alami, L. Redman, and Y. Stelate, ‘Mobile Technology for Improved Contraceptive Care in Morocco’, J Med Syst, vol. 45, no. 2, p. 16, Jan. 2021, doi: 10.1007/s10916-020-01684-6.
  19. :00-17:00, ‘ISO/IEC 25010:2011’, ISO. https://www.iso.org/fr/standard/35733.html (accessed Dec. 07, 2022).
  20. ‘ISO/IEC 25010 Software Quality Model - Codacy | Blog Developer’. https://blog.codacy.com/iso-25010-software-quality-model/ (accessed Dec. 07, 2022).
  21. :00-17:00, ‘ISO/IEC 25023:2016’, ISO. Accessed: Dec. 07, 2022. [Online]. Available: https://www.iso.org/fr/standard/35747.html
  22. L. Sardi, A. Idri, L. M. Redman, H. Alami, R. Bezad, and J. L. Fernández-Alemán, ‘Mobile health applications for postnatal care: Review and analysis of functionalities and technical features’, Comput Methods Programs Biomed, vol. 184, p. 105114, Feb. 2020, doi: 10.1016/j.cmpb.2019.105114.
  23. M. Bachiri, A. Idri, J. L. Fernández-Alemán, and A. Toval, ‘Mobile personal health records for pregnancy monitoring functionalities: Analysis and potential’, Comput Methods Programs Biomed, vol. 134, pp. 121–135, Oct. 2016, doi: 10.1016/j.cmpb.2016.06.008.
  24. A. Idri, M. Bachiri, and J. L. Fernández-Alemán, ‘A Framework for Evaluating the Software Product Quality of Pregnancy Monitoring Mobile Personal Health Records’, J Med Syst, vol. 40, no. 3, p. 50, Mar. 2016, doi: 10.1007/s10916-015-0415-z.
  25. S. Ouhbi, A. Idri, J. L. Fernández-Alemán, A. Toval, and H. Benjelloun, ‘Applying ISO/IEC 25010 on Mobile Personal Health Records’, presented at the International Conference on Health Informatics, Dec. 2022, pp. 405–412. Accessed: Dec. 07, 2022. [Online]. Available: https://www.scitepress.org/Link.aspx?doi=10.5220/0005216604050412
  26. K. Moumane, A. Idri, and A. Abran, ‘Usability evaluation of mobile applications using ISO 9241 and ISO 25062 standards’, SpringerPlus, vol. 5, no. 1, p. 548, Apr. 2016, doi: 10.1186/s40064-016-2171-z.
  27. A. Papageorgiou, M. Strigkos, E. Politou, E. Alepis, A. Solanas, and C. Patsakis, ‘Security and Privacy Analysis of Mobile Health Applications: The Alarming State of Practice’, IEEE Access, vol. 6, pp. 9390–9403, 2018, doi: 10.1109/ACCESS.2018.2799522.
  28. ‘Mobile apps security, vulnerabilities and common attacks’, Jun. 22, 2021. https://www.vaadata.com/blog/how-to-strengthen-the-security-of-your-mobile-applications-to-counter-the-most-common-attacks/ (accessed Jan. 24, 2023).
  29. P. A. H. Williams and A. J. Maeder, ‘“Security and Privacy Issues for Mobile Health”’, in Mobile Health: A Technology Road Map, S. Adibi, Ed. Cham: Springer International Publishing, 2015, pp. 1067–1088. doi: 10.1007/978-3-319-12817-7_44.
  30. K. Theodos and S. Sittig, ‘Health Information Privacy Laws in the Digital Age: HIPAA Doesn’t Apply’, Perspect Health Inf Manag, vol. 18, no. Winter, p. 1l, Dec. 2020.
  31. W. N. Price and I. G. Cohen, ‘Privacy in the Age of Medical Big Data’, Nat Med, vol. 25, no. 1, pp. 37–43, Jan. 2019, doi: 10.1038/s41591-018-0272-7.
  32. S. Maeen and S. Zykov, ‘Towards Social Network – Integrated E-Health: Identify User Attitudes’, Procedia Computer Science, vol. 55, pp. 1174–1182, Jan. 2015, doi: 10.1016/j.procs.2015.07.091.
  33. J. R. Oviatt and S. M. Reich, ‘Pregnancy posting: exploring characteristics of social media posts around pregnancy and user engagement’, Mhealth, vol. 5, p. 46, Oct. 2019, doi: 10.21037/mhealth.2019.09.09.
  34. R. L. Dekker, S. King, and K. Lester, ‘Social Media and Evidence-Based Maternity Care: A Cross-Sectional Survey Study’, J Perinat Educ, vol. 25, no. 2, pp. 105–115, 2016, doi: 10.1891/1058-1243.25.2.105.
  35. M. Smith, A. S. Mitchell, M. L. Townsend, and J. S. Herbert, ‘The relationship between digital media use during pregnancy, maternal psychological wellbeing, and maternal-fetal attachment’, PLOS ONE, vol. 15, no. 12, p. e0243898, Dec. 2020, doi: 10.1371/journal.pone.0243898.
  36. K. Moumane and A. Idri, ‘A Systematic Map of Mobile Software Usability Evaluation’, in Trends and Advances in Information Systems and Technologies, Cham, 2018, pp. 58–67. doi: 10.1007/978-3-319-77712-2_6.
  37. A. Puspaningrum, S. Rochimah, and R. Akbar, ‘Functional Suitability Measurement using Goal-Oriented Approach based on ISO/IEC 25010 for Academics Information System’, Journal of Information Systems Engineering and Business Intelligence, vol. 3, p. 68, Oct. 2017, doi: 10.20473/jisebi.3.2.68-74.
  38. N. A. Hasanah, L. Atikah, and S. Rochimah, ‘Functional Suitability Measurement Based on ISO/IEC 25010 for e-Commerce Website’, in 2020 7th International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), Sep. 2020, pp. 70–75. doi: 10.1109/ICITACEE50144.2020.9239194.
  39. M. Rodríguez, J. R. Oviedo, and M. Piattini, ‘Evaluation of Software Product Functional Suitability: A Case Study’.
  40. M. Nguyen Duc, T. Huynh Quyet, and H. Nguyen, ‘Improve the Performance of Mobile Applications Based on Code Optimization Techniques Using PMD and Android Lint’, Nov. 2016, pp. 343–356. doi: 10.1007/978-3-319-49046-5_29.
  41. M. Hort, M. Kechagia, F. Sarro, and M. Harman, ‘A Survey of Performance Optimization for Mobile Applications’, IIEEE Trans. Software Eng., vol. 48, no. 8, pp. 2879–2904, Aug. 2022, doi: 10.1109/TSE.2021.3071193.
  42. P. S. Kochhar, ‘Testing and debugging: A reality check’.
  43. G. Lamprinakos et al., ‘Using FHIR to develop a healthcare mobile application’, Nov. 2014, pp. 132–135. doi: 10.1109/MOBIHEALTH.2014.7015927.
  44. R. Saripalle, C. Runyan, and M. Russell, ‘Using HL7 FHIR to achieve interoperability in patient health record’, Journal of Biomedical Informatics, vol. 94, p. 103188, Jun. 2019, doi: 10.1016/j.jbi.2019.103188.
  45. A. Chatterjee, N. Pahari, and A. Prinz, ‘HL7 FHIR with SNOMED-CT to Achieve Semantic and Structural Interoperability in Personal Health Data: A Proof-of-Concept Study’, Sensors, vol. 22, no. 10, Art. no. 10, Jan. 2022, doi: 10.3390/s22103756.
  46. J. Yoo et al., ‘Development of an Interoperable and Easily Transferable Clinical Decision Support System Deployment Platform: System Design and Development Study’, Journal of Medical Internet Research, vol. 24, no. 7, p. e37928, Jul. 2022, doi: 10.2196/37928.
  47. I. Neamtiu and T. Dumitraş, ‘Cloud software upgrades: Challenges and opportunities’, in 2011 International Workshop on the Maintenance and Evolution of Service-Oriented and Cloud-Based Systems, Sep. 2011, pp. 1–10. doi: 10.1109/MESOCA.2011.6049037.
  48. B. A. Alzahrani, ‘Secure and Efficient Cloud-based IoT Authenticated Key Agreement scheme for e-Health Wireless Sensor Networks’, Arab J Sci Eng, vol. 46, no. 4, pp. 3017–3032, Apr. 2021, doi: 10.1007/s13369-020-04905-9.
  49. R. Sivan and Z. A. Zukarnain, ‘Security and Privacy in Cloud-Based E-Health System’, Symmetry, vol. 13, no. 5, Art. no. 5, May 2021, doi: 10.3390/sym13050742.
  50. M. Ali, M. E. Joorabchi, and A. Mesbah, ‘Same App, Different App Stores: A Comparative Study’, in 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft), May 2017, pp. 79–90. doi: 10.1109/MOBILESoft.2017.3.
  51. Y. Cheon and C. Chavez, ‘Converting Android Native Apps to Flutter Cross-Platform Apps’, in 2021 International Conference on Computational Science and Computational Intelligence (CSCI), Dec. 2021, pp. 1898–1904. doi: 10.1109/CSCI54926.2021.00355.
  52. K. Vishal and A. S. Kushwaha, ‘Mobile Application Development Research Based on Xamarin Platform’, in 2018 4th International Conference on Computing Sciences (ICCS), Aug. 2018, pp. 115–118. doi: 10.1109/ICCS.2018.00027.
  53. S. Vilkomir, K. Marszalkowski, C. Perry, and S. Mahendrakar, ‘Effectiveness of Multi-Device Testing Mobile Applications’, May 2015. doi: 10.1109/MobileSoft.2015.12.
  54. I. Santos, J. C. C. Filho, and S. R. S. Souza, ‘A survey on the practices of mobile application testing’, in 2020 XLVI Latin American Computing Conference (CLEI), Oct. 2020, pp. 232–241. doi: 10.1109/CLEI52000.2020.00034.
  55. K. S. Arif and U. Ali, ‘Mobile Application testing tools and their challenges: A comparative study’, in 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Jan. 2019, pp. 1–6. doi: 10.1109/ICOMET.2019.8673505.
  56. A. Méndez Porras, C. U. Quesada López, and M. Jenkins Coronas, Automated Testing of Mobile Applications: A Systematic Map and Review. 2015. Accessed: Feb. 26, 2023. [Online]. Available: https://kerwa.ucr.ac.cr/handle/10669/77115
  57. J. Klonovs, C. K. Petersen, H. Olesen, and A. Hammershoj, ‘ID Proof on the Go: Development of a Mobile EEG-Based Biometric Authentication System’, IEEE Vehicular Technology Magazine, vol. 8, no. 1, pp. 81–89, Mar. 2013, doi: 10.1109/MVT.2012.2234056.
  58. H. Liu, R. G. Crespo, and O. S. Martínez, ‘Enhancing Privacy and Data Security across Healthcare Applications Using Blockchain and Distributed Ledger Concepts’, Healthcare, vol. 8, no. 3, Art. no. 3, Sep. 2020, doi: 10.3390/healthcare8030243.
  59. Jovarie, ‘Importance of Technology in Tracing History’, Importance of Technology, Aug. 13, 2020. https://importanceoftechnology.net/importance-of-technology-in-tracing-history/ (accessed Dec. 07, 2022).
  60. D. J. Lu, M. Girgis, J. M. David, E. M. Chung, K. M. Atkins, and M. Kamrava, ‘Evaluation of Mobile Health Applications to Track Patient-Reported Outcomes for Oncology Patients: A Systematic Review’, Adv Radiat Oncol, vol. 6, no. 1, p. 100576, Oct. 2020, doi: 10.1016/j.adro.2020.09.016.
  61. N. Bouri and S. Ravi, ‘Going Mobile: How Mobile Personal Health Records Can Improve Health Care During Emergencies’, JMIR Mhealth Uhealth, vol. 2, no. 1, p. e8, Mar. 2014, doi: 10.2196/mhealth.3017.
  62. Y. Choi et al., ‘Development of a Mobile Personal Health Record Application Designed for Emergency Care in Korea; Integrated Information from Multicenter Electronic Medical Records’, Applied Sciences, vol. 10, no. 19, Art. no. 19, Jan. 2020, doi: 10.3390/app10196711.
  63. P. T. Low, C. G. Ng, M. S. Kadir, and S. L. Tang, ‘Reminder through mobile messaging application improves outpatient attendance and medication adherence among patients with depression: An open-label randomised controlled trial’, Med J Malaysia, vol. 76, no. 5, pp. 617–623, Sep. 2021.
  64. C. Horsch et al., ‘Reminders make people adhere better to a self-help sleep intervention’, Health Technol (Berl), vol. 7, no. 2, pp. 173–188, 2017, doi: 10.1007/s12553-016-0167-x.
  65. J. Leight et al., ‘The effects of text reminders on the use of family planning services: evidence from a randomised controlled trial in urban Mozambique’, BMJ Global Health, vol. 7, no. 4, p. e007862, Apr. 2022, doi: 10.1136/bmjgh-2021-007862.