Exploring User Adoption and Experience of AutoML Platforms: Learning Curves, Usability, and Design Considerations

doi:10.21203/rs.3.rs-4708019/v1

Download PDF

Article

Exploring User Adoption and Experience of AutoML Platforms: Learning Curves, Usability, and Design Considerations

https://doi.org/10.21203/rs.3.rs-4708019/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Human daily activities and businesses generate a high volume of data which are expected to be transformed for the benefit of businesses and mankind. Organizations make use of machine learning platforms to make informed decisions from well gleaned insights of their real-time data. The process of learning machine learning is seemingly not an easy one, making it tedious for employees to learn easily and quickly. Meanwhile, the introduction of automated machine learning (AutoML) has made this easier. However, it is essential to understand how users embrace and implement the AutoML platform for their real-world problems. To achieve this, we conducted a quantitative study with 38 users focusing on understanding firstly, the learning curve (i.e. the alignment of users’ performance proportionately with the time required to complete a given machine learning task at any given time) and experience of people in the process of learning machine learning. Secondly, the strengths and weaknesses in the design and usability of ML and AutoML. Thirdly, the gaps in the usage or user experience of a newbie - an inexperienced or fresh entrant in the machine learning domain- an inexperienced and fresh entrant in the machine learning domain and professionals, and fourthly, the design factors needed to improve the user experience. Our findings revealed that users have high expectations in the usability of AutoML. In this study, we were able to carry out an assessment of awareness rate of AutoML among the respondents, users’ learning curve in AutoML environment, usability assessment as it relates to variations in users of AutoML, human computer interactivity in relation to in terms of user centeredness and experience of the environment. The study revealed the level of awareness, reasons for apathy and some usability concerns begging for improvement to attract a high rate of usefulness and adoption. In the near future, we hope to take this work further by engaging frequent users of various AutoML environments to ascertain the level of satisfaction using such platforms and identify areas of concern.

Biological sciences/Computational biology and bioinformatics/Data mining

Biological sciences/Computational biology and bioinformatics/Data processing

Biological sciences/Computational biology and bioinformatics/Machine learning

Machine Learning

AutoML

Quantitative Study

Learning Usability

User Experience

Awareness Rate

Human Computer Interactivity

In the world today, every facet of human lives and businesses generate high volumes of data daily and it is imperative for organizations to make good decisions without human intervention, and to do it as fast and accurately as possible [1]. Most industries working with large amounts of data have recognized the value of machine learning technology in gleaning insights from their real-time data, making them more productive, efficient, and competitively relevant [2]. Machine Learning (ML) is an offshoot of the field of Computer Science and Artificial Intelligence that focuses on the use of data, algorithms, and statistical techniques to mimic human reasoning. It imitates how humans learn and solve complex tasks.

The usage of Machine Learning algorithms for an application or a task often require coding skills or experience. Machine Learning is a major and critical field expected to drive the 4th Industrial Revolution. Its need and demand are high and cuts across diverse fields. As a result, various individuals, both tech and non-tech, are desirous to learn machine learning. A major obstacle to many of the newbies and non-technical people to acquiring the machine learning skills is the difficulty that the existing programming languages used for ML and their integrated development environment presents [1]. Meanwhile, the core of user experience is usability and productivity, and not just the positive emotions derived from the use of a tool or product [3], Recently, more visual, and easier to use platforms such as Google AutoML, Azure AutoML, AutoKeras, AutoScikit, bigML among others have been developed to assist ML enthusiasts, newbies and even professionals to carry out diverse ML tasks [4].

Automated Machine Learning (AutoML) involves making machine learning accessible to non-machine learning experts by automating the application of machine learning models to real-world problems [5]. This is crucial because the scope of ML has expanded beyond the realm of computing alone and is now widely used and relied upon in other disciplines such as medical sciences, business, environmental sciences, chemical sciences, natural sciences, agriculture, and arts [4]. ML has become a multidisciplinary process. The applications of machine learning are limitless, spanning from healthcare to transportation, agriculture, marketing, manufacturing, financial services, government, and cybersecurity [6-9].

1.1 Research Aim and Questions

The aim of this research is to understand the user experience regarding the use of AutoML. The research questions are as follows:

Are there gaps in using generic ML environments such as Python SDK, Matlab?
What strengths exist in the design and usability of platforms such as AutoML recently developed to ease the learning of ML or operations/application of ML?
What weaknesses exist that can be further improved upon?

To answer research question (i), we engaged users with experience in using generic machine learning environments such as Python. Users involved in answering research questions (ii) and (iii) were limited to those with experience in AutoML.

2.1 The AI-HCI Concept

Artificial Intelligence (AI) is a field of Computer Science that is concerned with developing machines that are smart and capable of performing complex tasks that typically would have required human intelligence [10, 11]. AI is built to mimic human reasoning, intuition, and discernment.

Human Computer Interaction (HCI) is a field of study that centers on optimizing how users and computing devices (as well as applications) interact by designing interactive interfaces that satisfy the needs of the users and give a memorable user experience [12]. It is a multidisciplinary field of computer science that is significant for the success of companies that build software.

Artificial Intelligence (AI) is a field of Computer Science that focuses on developing intelligent machines capable of performing complex tasks that traditionally required human intelligence [10, 11]. AI aims to emulate human reasoning, intuition, and discernment. Human-Computer Interaction (HCI) is a multidisciplinary field that optimizes the interaction between users and computing devices or applications by designing interactive interfaces that meet user needs and provide a positive user experience. It plays a crucial role in the success of software companies [12].

According to [13], AI and HCI are two computing fields united by a common focus. AI systems, whether software or hardware, are designed for human use. Therefore, establishing a strong relationship between AI and HCI is essential. When developing AI applications, giving attention to human-computer interaction, including usability and user experience, is critical in designing AI applications that align with specific user needs. This user-oriented approach to AI is known as the user-oriented approach (Abolkasim, 2021). The increasing use of AI technologies in user-oriented applications has emphasized the need to study the relationship between these two fields [14].

2.2 The Concept of Machine Learning

Machine Learning is the process of using mathematical models of data and statistical learning theory to aid a computer to learn and continue learning and improving, based on experience. Apart from the use of mathematical models and statistical theory, it is also imperative for machine learning experts to be proficient in some programming languages (e.g., Python and R), to solve many challenging tasks across several domains. It is worthy to note that building a machine learning solution still requires substantial resources as well as human expertise (Sun et al., 2023).

To build an ML model, the process must go through the following steps: (Yip, 2020)

Data Collection
Data Preprocessing
Model Building and Training
Model Evaluation
Model Deployment

These steps require human experts to navigate through with their programming proficiency as seen in figure 1.

As organizations are more involved in predictive analytics and AI applications development for insights and good decisions (more than ever before), it has become imperative for both tech and non-tech to know machine learning, but the usage of ML algorithms for a task often requires coding skills. Because learning this skill requires substantial resources for learning, automated machine learning (AutoML) was created to ease the difficulties of non-experts in using ML to achieve productivity (Sun et al., 2023).

2.3 Automated Machine Learning (AutoML)

Automated machine learning (AutoML) is the process of automating the selection, parameterization, and composition of machine learning models. These models are then applied to real-world problems using automation. According to Felix Mohr & Marcel Wever (2022), AutoML is developed to solve the problem of automatically finding the pipeline (i.e. pre-processing and learning algorithms and the hyperparameters) with the best generalisation performance.

What AutoML does is to automate the tasks involved in the process of training machine learning models. Some of the hand-coded tasks necessary for human ML experts to do are data cleaning, preprocessing, feature selection, hyperparameter tuning, and model building. These tasks, which are tedious, time-consuming, and repetitive, become faster and can explore new approaches during modelling with the use of AutoML. It automates all manual tuning and modelling that human experts do to find optimal hyperparameters and models.

AutoML tools and platforms vary in terms of their features, complexity, and level of automation. Some platforms provide a user-friendly interface or graphical user interface (GUI) that allows users to specify their requirements and constraints. Others offer programmatic APIs or command-line interfaces (CLIs) for more advanced users to interact with the AutoML system. Additionally, some AutoML solutions leverage cloud computing resources to handle large-scale data and computationally intensive tasks.

The goal of AutoML is to democratise machine learning by automating repetitive tasks, reducing the need for extensive domain expertise, and enabling users to focus more on problem formulation and interpretation of results. However, it is important to note that AutoML is not a one-size-fits-all solution and may not always outperform expert-designed machine learning pipelines. It should be seen as a tool that complements human expertise and enables faster iteration and experimentation in the machine learning workflow.

AutoML systems typically use a combination of machine learning algorithms, optimization techniques, and search strategies to automate the various tasks involved in model building. They are designed to learn from historical data to improve the accuracy and speed of the model building process. There are two main types of AutoML:

Fully Automated Machine Learning: In this approach, the entire process of model building is automated, including data preparation, feature engineering, algorithm selection, hyperparameter tuning, and model deployment. The user simply provides the input data and specifies the problem to be solved, and the system takes care of the rest.
Semi-Automated Machine Learning: In this approach, the user retains some control over the model building process. The system automates certain tasks such as feature engineering and hyperparameter tuning, but the user can still specify the algorithm to be used and adjust certain parameters manually.

2.4 Human-centred AutoML

While the HCI community has proposed guidelines and strategies for designing interactive user interfaces for AI applications for over two decades, the implementation of these guidelines has gained more emphasis in recent times [14]. A key approach in interaction design is user-centred design [12]. User-centered design involves gaining insights about users' experiences through usability testing [12, 15]. A tool, product, or software is considered user-centered if its design phase revolves around gaining a deep understanding of the product's users and their motivations for using it. When the real issues faced by users are understood and addressed prior to creating a product for them, the product can be considered user centered. As highlighted in the blog post titled "Rethinking AutoML: Advancing from a Machine-Centered to Human-Centered Paradigm" by [16], the initial AutoML approaches did not adequately consider the diverse categories of users (see figure 2).

As shown in Figure 2, a user-centred AutoML is tailored to the needs and established workflows of users (which in this case are practitioners), hence supporting them in their daily business operations in the most effective and efficient way [17]. Some of the AutoML solutions are Auto-Sklearn, Google Cloud AutoML, H2O AutoML, AutoKeras, Amazon Lex.

2.5 Gaps in the Usage of AutoML

Existing studies on AutoML have identified several limitations and drawbacks associated with its usage. One significant drawback is that AutoML is often considered a black box solution, making it challenging for users to understand how the models arrive at their predictions [18]. Another limitation is the lack of intermediate results provided to users, which hinders their ability to evaluate the system effectively. Additionally, AutoML solutions typically do not allow for human-in-the-loop interactions, further restricting user control and involvement in the model development process [17]. Addressing these limitations would contribute to democratizing machine learning, increasing accessibility, and promoting innovation and business efficiency.

To understand user’s experience on the use of Generic ML environments and AutoML environments from the angle of newbies and experienced users, we conducted a user study with users composed of

ML practitioners - category of users who handles the business aspect of Machine learning and relates with the stakeholders on the possibilities and strengths of engaging in goal-driven ML tasks for social good,
ML Enthusiasts - persons passionate about machine learning and what outcome(s) comes from it; these categories of users may not necessarily be experts or professionals in ML, and newbies in ML [19].

This study employs a quantitative approach of study using questionnaires. In designing the questionnaire, we adopted the system's usability scale [20] and ISO Usability metrics [21] to measure users' experience. The questionnaire has 5 sections. The first section focused on demographic information, experience and learning curve in acquiring ML skills using a generic environment such as Python. After section 1, participants who had no experience in the use of AutoML environment were asked to state expectations and design suggestions on how AutoML can ease the learning curve of acquiring machine learning skills. For participants with AutoML experience, section 3 and 4 focused on knowing their usability experience and performance evaluation of the AutoML. Finally, they gave suggestions on how AutoML can be further improved based on their experience. A summary of the questionnaire structure and prompt is shown in Figure 3.

3.1 Recruitment and Data Collection

Recruitment was done via sending messages across various social media groups and communities of HCI and AI professionals. Such communities include a University in South-west Nigeria, AI Saturday Club, Data Scientists Network, and Women in AI. In addition, the snow-ball approach was also adopted in recruitment of participants. Participation was voluntary and an ethical clearance was obtained from an appropriate institution in Nigeria. The recruitment lasted for 9 months (September 2022 - April 2023). It was difficult getting users in Nigeria who have experience in the use of autoML. So, the recruitment and study took longer than envisaged.

3.2 Data Analysis

During the initial phase of the study, 38 participants who possessed experience in the conventional ML platform environment were involved. Subsequently, the second phase concentrated on examining user experience within the AutoML environment, and out of the initial 38 participants, 16 individuals had prior exposure to AutoML and actively engaged in the usability evaluation process. Nielsen's guideline (1994) suggesting a minimum of 3-5 users for a usability study was taken into consideration, affirming the suitability of proceeding with the 16 selected participants to identify and address usability concerns specific to AutoML.

In the first phase of the study, which aimed to investigate user experience in a generic environment, a total of 38 participants actively participated. Among the participants, 29.95% were female and 71.05% were male. In terms of ML experience, 55.3% had 1-2 years of experience, 39.5% had 3-4 years, and 5.3% had 5-6 years. Categorizing the users, 39.5% identified as ML Experts, 36.8% as ML Enthusiasts, 18.4% as Newbies, and 5.3% fell under the category of "other.". Moving on to the second phase of the study, which focused specifically on user experience with AutoML, a total of 16 participants took part. Among these participants, 12.5% were female and 87.5% were male. In terms of ML experience, 37.5% had 1-2 years, 56.25% had 3-4 years, and 6.25% had 5-6 years.

4.1 Phase 1: Exploratory Analysis

In this section, we discussed insights gained from the analysis of the demographic information vis-a-vis their expectation from autoML or general ML experience. Three key factors emerged prominently from this exploratory analysis, which are discussed as follows:

4.1.1 Learning Curve

One notable observation was that individuals with previous programming experience found it comparatively easier to learn machine learning. On the other hand, participants without prior programming experience faced difficulties in grasping machine learning concepts. Figure 4 illustrates the results, indicating that the introduction of AutoML could potentially alleviate the learning curve associated with machine learning. Out of the participants, 27 individuals (77.1%) agreed that AutoML could facilitate the learning process, while 8 participants (22.9%) expressed disagreement. This highlights the high expectations placed on AutoML in terms of its usability by the respondents.

4.1.2 Apathy Issues

AutoML, which aims to address the AI skills gap, has encountered a lack of enthusiasm among university students in Nigeria, especially those attending universities in the Southwestern region. Our study uncovered a low level of experience in AutoML among the respondents, largely due to a lack of awareness. Figure 5 highlights this issue, with 60% of the participants indicating that they are hearing about AutoML for the first time.

4.1.3 Expectation from AutoML

We conducted further analysis on participants who had no prior experience in using AutoML. We wanted to understand their willingness to use AutoML if given access or exposure, as well as their expectations and design suggestions for the platform. The results, shown in Figure 6, indicate that 92.9% of the participants have positive expectations from using an AutoML tool to enhance their performance in machine learning projects. This high positive response rate can be attributed to the potential for increased productivity and reduced reliance on human intervention in machine learning projects, as highlighted by [22].

When asked about design suggestions, the participants emphasized the importance of simplicity and ease of understanding and interpreting results. They suggested considering the assessment quality of unsupervised learning results and incorporating an interface for data uploading. Additionally, they highlighted the need for the output of trained data to be presented in a way that even machine learning newbies can comprehend without extensive assistance from ML professionals. Furthermore, an analysis of the participants' programming language preferences revealed that 76.32% of them started their programming journey with Python. Based on this, we recommend that AutoML platforms be designed in a way that can generate Python code. This would enable newbies to view and learn from the underlying code behind their AutoML projects, further enhancing their understanding and learning experience.

4.2 Phase 2: Exploratory Analysis

In this phase, we focused on participants who had prior experience with AutoML to gather insights into their usability experience. For those who did not have experience with AutoML, we organized a 3-day workshop dedicated to AutoML. The results and findings from that workshop will be published in a future article. In this section, we will discuss the insights gained from analyzing the demographic information in relation to the usability experience of those participants who had prior experience with AutoML.

4.2.1 Gender Issues in Adoption of autoML

Our study revealed a significant gender disparity among the respondents, with males accounting for 87.5% and females comprising only 12.5%. Despite the distribution and administration of questionnaires to both genders, this outcome reflects a consistent trend of low technology adoption by females, which aligns with concerns raised by [23]. The underrepresentation of females in automated environments, where they make up less than 11%, puts them at a higher risk of job loss due to automation. Therefore, it is crucial for developers and usability testers to prioritize gender-friendly and inclusive programming environments that cater to individuals from diverse backgrounds. Considering gender diversity in the design and development of AutoML platforms, as emphasized by [24], becomes highly imperative in addressing this disparity.

4.2.2 Rigidity

Our findings also revealed that a significant number of respondents who identified themselves as programming experts do not utilize AutoML. Instead, they prefer the traditional approach of running and evaluating their machine learning models. This sentiment aligns with concerns raised by Naik (2022), who highlights that experts perceive AutoML as too restrictive, as it only handles a small portion of the end-to-end process. According to Naik, the most impactful task in machine learning, which is feature engineering, is domain-specific and cannot be fully automated. Consequently, AutoML is seen as a job generator that facilitates the execution of a series of experiments based on a predefined framework.

4.3 Usability Issues of AutoML

Usability is a measure of how well a specific user (Newbie, ML expert, ML enthusiast or Domain expert) can use a product or design (in this case, AutoML) to achieve a defined goal effectively, efficiently, and satisfactorily (reference). This section was targeted at users who have autoML experience. Some common autoML platforms used are BigML and, Rapid Miner. A total of 16 participants with autoML experience took part in this section. Figure 7 shows a distribution of the years of experience of using autoML.

Usability scale [20] and ISO Usability [21] metrics were adopted to evaluate usability issues on autoML. A 5-likert scale was used for this measure. Details uncovered are discussed as follows:

4.3.1 Ease of Use and Usage Frequency

Figure 8 provides insights into participants' perceptions of the ease of use and frequency of usage of AutoML. Regarding ease of use, 23.07% of participants strongly agreed that AutoML is easy to use, 53.8% agreed, 15.4% were neutral, and 7.8% strongly disagreed. In terms of frequency of usage, 25% of users strongly agreed that they like to use AutoML frequently, 37.5% agreed, 31.25% were neutral, and 6.25% strongly disagreed. Interestingly, we observed a similar distribution on the extreme ends of the scale for both ease of use and usage frequency, but some differences in the middle range. This indicates that while ease of use is important, it does not necessarily guarantee that users will always prefer to use the system frequently. Upon further investigation, these differences in frequency and ease of usage could be attributed to factors such as limited feature engineering modules and use cases, as well as limitations in the explainability and interpretability of AutoML. Furthermore, an analysis of the participants' programming language preferences revealed that 76.32% of them started their programming journey with Python. Based on this, we recommend that AutoML platforms be designed in a way that can generate Python code. This would enable newbies to view and learn from the underlying code behind their AutoML projects, further enhancing their understanding and learning experience.

4.3.2 Technical Support

Figure 9 displays the participants' responses regarding their need for technical support and the requirement to learn a lot to use AutoML effectively. The results indicate that 12.5% strongly disagreed, 31.25% disagreed, 37.5% were neutral, and 18.75% agreed with the need for technical support. Similarly, 18.75% strongly disagreed, 25% disagreed, 37.5% were neutral, and 18.75% agreed with the requirement to learn a lot before using AutoML. Interestingly, the ratio of participants who agreed and those who were neutral is the same for both the need for technical support and the need to learn a lot. This suggests that some existing AutoML platforms still require users to have a certain level of technical knowledge to operate effectively. This finding contradicts the original purpose of AutoML, which aimed to simplify the process. Therefore, further research at the intersection of AI and HCI is needed to address this technicality and make AutoML more user-friendly. These findings align with the results of [25], which revealed that many current AutoML systems still rely heavily on human intervention.

4.3.3 Confidence

According to Sakpere (2019), confidence is key to continuation of a task or engaging a new environment/learning paradigm. As a result, in this research we sought to understand the confidence level of users in using autoML. 18.75% strongly agreed to a high level of confidence using autoML, 62.5% agreed, 12.5% were indifferent and 6.25% disagreed.

4.3.4 User Experience

From the responses gathered, users' experiences with AutoML can be summarized as follows:

Easy understanding of machine learning concepts without coding: Users found that AutoML provided a simplified approach to grasp the fundamental concepts of machine learning without the need for extensive coding knowledge.
Elimination of complexity in machine learning project workflow: Users appreciated how AutoML streamlined the workflow associated with machine learning projects, removing the complexities and making the process more accessible and manageable.
Development of models in a short time with minimal effort: Users expressed satisfaction with the efficiency of AutoML, as it allowed them to develop models within a short timeframe and with reduced effort compared to traditional approaches.

Flexibility in using different learning algorithms: Users found it intriguing that AutoML offered the flexibility to experiment with various learning algorithms without requiring significant changes to the underlying code structure. This flexibility enhanced their ability to explore different approaches and optimize their models.

Overall, users had positive experiences with AutoML, highlighting its ability to simplify the understanding of machine learning concepts, streamline project workflows, and provide efficient model development with flexibility in algorithm selection.

4.4 Learning via AutoML

The findings as from the survey regarding participants' opinions on AutoML and its role in learning machine learning which is summarized in table 1 is explained as follows:

General perception of AutoML for learning ML: 12% of the participants strongly agreed that autoML is an ideal tool for learning ML, while 37.5% agreed, 18.75% were neutral, 12% disagreed, and 18.75% strongly disagreed. This indicates a mixed opinion among users regarding the suitability of autoML for learning ML.
Comparison of autoML with Python for learning ML: When specifically comparing autoML to Python, 18.75% of participants strongly agreed that autoML is easier to use for learning ML, 43.75% agreed, 6.25% were neutral, 12.5% disagreed, and 18.75% strongly disagreed. These results suggest that while some participants find autoML more user-friendly than Python, there are still others who prefer traditional programming languages like Python for learning ML.
Combination of autoML with a programming language for learning ML: Regarding the combination of autoML with a programming language for learning ML, 37.5% of participants strongly agreed that autoML is best used in conjunction with a programming language, another 37.5% agreed, 12.5% were neutral, 6.25% disagreed, and 6.25% strongly disagreed. This indicates that many participants recognize the value of combining autoML with a programming language to enhance their learning experience.
AutoML suitability for newbie: With respect to recommendation of autoML for newbies to learn machine learning, 6.25% of participant strongly agreed that autoML is suitable for newbies to learn machine learning, 43.75% agreed, 6.25% were neutral, 31.25% disagreed, and 12.5% strongly disagreed. This implies that many participants do not perceive autoML has a platform for newbies to learn the intricacies of machine learning coding.
AutoML application domain suitability: We further investigated if autoML is either suitable for business or educational purposespurpose. 18.75% strongly agreed autoML is suitable for business purposes and not educational purposespruposes, another 43.75% agreed, 18.75% are indifferent, 12.5% disagree and 6.25% agree.

Table 1: Research Questions on the contribution of AutoML to Learning ML

Insight Questions	Strongly Agree ( %)	Agree ( %)	Neutral ( %)	Disagree ( %)	Strongly Disagree ( %)
Is AutoML ideal for Learning?	12	37.5	18.75	12	18.75
Is AutoML better than Python?	18.75	43.75	6.25	12.5	18.75
Should AutoML be combined with other Programming Language?	37.5	37.5	12.5	6.25	6.25
I will encourage a newbie to use autoML for learning machine learning techniques	6.25	43.75	6.25	31.25	12.5
AutoML is best suitable for developing business solutions and not necessarily for educational/learning purposes	18.75	43.75	18.75	12.5	6.25

4.5 Evaluation of Computational Capabilities of Cloud-Based/autoML Tools Compared to Generic ML Tools

The evaluation of the computational capabilities of cloud-based/autoML tools compared to generic ML tools can be summarized as follows:

Training Time on cloud-based Tools is faster

Out of 16 respondents, 31.3% of them agreed while 43.8% strongly agreed that the training time on cloud-based tools is faster. But 12.5% of them disagreed and another 12.5% were indifferent/neutral. This is summarized on table 2

Reproducibility is Better Using Cloud-based Tools

In the result of 16 responses, 13 participants (81.3%) agreed that reproducibility, which is the ability to replicate an ML operation carried out and getting the same results as the original work, is better using cloud-based tools, 2 participants (12.4%) strongly agreed, nobody disagreed, while 1 person was indifferent.

Table 2: Users’ Perception on Cloud-Based ML Tools as compared to Traditional ML Tools

Evaluation Metrics	Fast (%)	Faster (%)	Slow (%)	Indifferent (%)
Training Time	31.30	43.80	12.50	12.50
Reproducibility	81.30	12.50	0.00	6.30

Classification Algorithm Performs Better

Out of 16 responses who use classification algorithms on autoML, 6 participants (37.5%) agreed that classification algorithm in autoML performs better, which is the ability to correctly classify an object, 3 participants (18.8%) strongly agreed, 3 people disagreed, while 4 people were indifferent as shown in table 3.

Table 3: Users’ Perception on Speed of Cloud-Based ML Tools as compared to Traditional ML Tools

Evaluation Metrics	Fast (%)	Faster (%)	Slow (%)	Slower (%)	Indifferent (%)
Classification Performance	37.50	18.80	18.80	-	25.00
Confidentiality (Security)	37.50	18.80	31.30	-	12.50
Tuning & Optimization	18.80	6.30	25.00	25.00	25.00

Confidentiality (Security)

A recent research has expressed privacy concerns about the use of autoML (Sun et. al., 2020). These privacy concerns vary from leakage of training data to inference attack. As a result, in this research we sought to understand user’s experience and confidence with respect to their view of autoML's ability to ensure confidentiality of training data compared to other traditional platforms or approaches. Out of 16 responses, 6 participants agreed confidentiality of training data on autoML is better, 3 strongly agreed, 2 are indifferent and 5 disagree.

Hyperparameter Tuning and Optimization

In machine learning, hyperparameter tuning or optimization is needed to choose a set of optimal parameters for a learning algorithm. This process has often been regarded as tedious and the traditional brute force approach has not been efficient (Sandha et. Al., 2020). It's believed that autoML is useful in finding optimal hyperparameters. As a result, during the usability study, we sought to understand users' experience with respect to using autoML for finding an optimal hyper parameter especially when compared to traditional approach. 1 participant strongly agreed that the use of autoML performs better for finding optimal parameters, 3 people agreed, 4 were indifferent, 4 disagreed and 4 strongly disagreed.

4.6 Design Recommendations

After identifying usability issues in autoML, we proceeded to involve participants in providing suggestions for enhancing the design of autoML and the factors that should be considered during its development.

4.6.1. Enhancing Usability and Acceptance of AutoML: Design Insights

The insights gathered from participants regarding design suggestions to enhance the usability and widespread acceptance of AutoML can be grouped into the following key themes:

User Interface and Ease of Use

Participant 26 emphasizes the need for a user-friendly interface that simplifies the upload of data and the presentation of trained data. The aim is to enable individuals new to machine learning to comprehend and interpret training outcomes independently, without relying on professional assistance.

Participant 19 provides a comprehensive list of recommendations: Ensuring explainability and interpretability of the system, crafting a user-friendly interface, offering robust documentation, allowing customization, ensuring flexibility, and prioritizing privacy and security considerations.

Enhancements and Technical Adjustments

One participant, represented by Participant 31, suggests adopting a Python-based platform with necessary plugins. The ease of library installation should be clearly communicated, as challenging installations of machine learning libraries could discourage users.

A call for more advanced feature engineering modules is made by Participant 3. They highlight that existing AutoML solutions often lack comprehensive feature engineering capabilities, which are crucial for optimizing the performance of machine learning models. Additionally, Participant 13 emphasizes the importance of improving explainability and interpretability features within AutoML systems, underscoring the need for transparent model outcomes.

Accessibility and Scope

Participant 14 underscores the significance of evaluating the quality of unsupervised learning results, particularly in the absence of a benchmark for comparison. Participant 20 reflects on the limited range of use cases for AutoML due to the dynamic nature of the problems addressed in everyday scenarios.

These insights offer valuable guidance for refining the design of AutoML, enhancing its usability, and expanding its applicability across diverse contexts.

4.6.2 Considerations in AutoML Design

Participants further pointed out factors to be considered in autoML design. These are:

Handling Diverse Data Types

"Taking into account the prevalence of common data types, AutoML should also cater to complex data types to ensure versatility in its applications."

"Prioritize ease of use, comprehensive documentation, and well-commented program codes to enhance the user experience."

Synergy between HCI and AI Methodologies

The feedback or comments given is as follows:

Participant ID	Comments
P31	Implementing a robust user interface and an interactive platform is pivotal in elevating the user experience
P26	Promote clear comprehension and meaningful interpretation of trained data by focusing on intuitive design
P21	Center the design around optimizing the user experience.
P22	Yes, particularly for individuals less familiar with coding. This AutoML could significantly assist them in harnessing its capabilities
Unknown	Absolutely, considering the ongoing shift towards AI and data science in the realm of computing. Aligning HCI practices with this trend can result in an enhanced platform that instills confidence.

4.7 Limitation and Future Work

Our participants are basically Africans with the majority being male. We also have a larger percentage of the participants to be Machine Learning enthusiasts with intermediate level of programming. Also, our research did not indicate whether the participants are active users of AutoML or non-active users. Soon, we hope to take this work further by engaging frequent users of various AutoML environments to ascertain the level of satisfaction using such platforms and identify areas of concern.

In our study, participants suggested one of the design considerations to make AutoML more usable and widely acceptable is customizability, which is vital for real-world problem-solving. For example, “difficulty in installing machine learning libraries to make operations dynamic can put a user off”. “Currently, most AutoML have limited feature engineering modules to achieve better performance for machine learning models” (P31) (P19) (P3).Our research discovered from participants’ responses that AutoML is too ‘Blackboxed’. The whole process of operation within is not transparent (P26).Furthermore, we found out that checking intermediate results and communicating data and models with teams and stakeholders is not clear. “More effort should be placed on explainability and interpretability of the AutoML” (P13) (P19) (P26). From participants’ reports on the involvement of HCI in the AutoML/AI field, our study saw the need to address the problem of user experience by providing a good user interface and proper interactive platform to boost user experience (P21) (P31). Another problem that our study discovered is that AutoML is limited in operation in some domains. Part of participants’ responses is “I think AutoML has limited Use Cases due to the ever-changing problems we solve every day” (P20). Lastly, our research revealed that many users have high expectations for AutoML, with most participants agreeing that it could ease the learning process of machine learning. This underscores the potential of AutoML in bridging the AI skills gap and enhancing productivity. Future research can further explore user satisfaction and delve deeper into specific areas of concern by engaging frequent users of various AutoML environments.

Soon, we aim to expand our research by involving frequent AutoML users, seeking to understand their level of satisfaction and addressing their specific needs and concerns. By doing so, we can contribute to the continuous improvement and development of AutoML platforms to better meet the expectations and requirements of users.

Ethical Approval and Consent to Participate

The ethical approval number for this research is given as BUREC/COCCS/CSC/0002. The ethical clearance is attached to this submission.

We confirm that all methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

All the authors consent to submit and publish this research in the Scientific report.

Availability of supporting data

The datasets generated and/or analysed during the current study are available in the in this googledrive link: https://docs.google.com/spreadsheets/d/1Nq0NIRIhK7lKA42F3ecz4YAuNeIEvlpYP3oCFqZjHUY/edit?usp=sharing

Completing Interests

All the authors declare that there is no competing of interest in carrying out this research work. No complete interest in submitting the research work for publication.

Chiang, C.-W. and M. Yin. Exploring the Effects of Machine Learning Literacy Interventions on Laypeople’s Reliance on Machine Learning Models. in 27th International Conference on Intelligent User Interfaces. 2022.
Haleem, A., et al., Artificial intelligence (AI) applications for marketing: A literature-based study. International Journal of Intelligent Networks, 2022.
Unsöld, M., Measuring Learnability in Human-Computer Interaction. 2018, Ulm University.
Sun, Y., et al. AutoML in The Wild: Obstacles, Workarounds, and Expectations. in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.
Arzani, B., K. Hsieh, and H. Chen. Interpretable Feedback for AutoML and a Proposal for Domain-customized AutoML for Networking. in Proceedings of the Twentieth ACM Workshop on Hot Topics in Networks. 2021.
Akinwunmi, A.O., et al., Geospatial Cloud-Based Model for Mitigating Impact of Natural Disaster and Security Threats using Smart Digital Devices. International Journal of Wireless and Microwave Technologies (IJWMT), 2023. 13(2): p. 23-36.
Ajagbe, S.A., et al., Multi-classification of Alzheimer disease on magnetic resonance images (MRI) using deep convolutional neural network (DCNN) approaches. International Journal of Advanced Computer Research, 2021. 11(53): p. 51.
Arogundade, O., et al. Improved predictive system for soil test fertility performance using fuzzy rule approach. in Soft Computing and its Engineering Applications: Second International Conference, icSoftComp 2020, Changa, Anand, India, December 11–12, 2020, Proceedings 2. 2021. Springer.
Cioffi, R., et al., Artificial intelligence and machine learning applications in smart production: Progress, trends, and directions. Sustainability, 2020. 12(2): p. 492.
BOUCHER, P.N., Artificial intelligence: How does it work, why does it matter, and what can we do about it? 2020.
Husbands, P., et al., Artificial evolution: A new path for artificial intelligence? Brain and cognition, 1997. 34(1): p. 130-159.
Rogers, Y., H. Sharp, and J. Preece, Interaction design: beyond human-computer interaction. 2023: John Wiley & Sons.
Grudin, J., AI and HCI: Two fields divided by a common focus. AI magazine, 2009. 30(4): p. 48-48.
Amershi, S., et al. Guidelines for human-AI interaction. in Proceedings of the 2019 chi conference on human factors in computing systems. 2019.
Sakpere, A.B., A.V. Kayem, and T. Ndlovu. A usable and secure crime reporting system for technology resource constrained context. in 2015 IEEE 29th International Conference on Advanced Information Networking and Applications Workshops. 2015. IEEE.
Marius, L.A., Tornede. Rethinking AutoML: Advancing from a Machine-Centered to Human-Centered Paradigm. 2022 [cited 2023 3rd Feb.]; Available from: https://www.automl.org/rethinking-automl-advancing-from-a-machine-centered-to-human-centered-paradigm/.
Pfisterer, F., J. Thomas, and B. Bischl, Towards human centered AutoML. arXiv preprint arXiv:1911.02391, 2019.
Zhou, Z., et al., ML lifecycle canvas: designing machine learning-empowered UX with material lifecycle thinking. Human–Computer Interaction, 2020. 35(5-6): p. 362-386.
Bhattacharya, A., Applied Machine Learning Explainability Techniques: Make ML models explainable and trustworthy for practical applications using LIME, SHAP, and more. 2022: Packt Publishing Ltd.
Saenz, J., et al. The usability analysis of chatbot technologies for internal personnel communications. in IIE Annual Conference. Proceedings. 2017. Institute of Industrial and Systems Engineers (IISE).
Georgsson, M. and N. Staggers, Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. Journal of the American Medical Informatics Association, 2016. 23(1): p. 5-11.
Crisan, A. and B. Fiore-Gartland. Fits and starts: Enterprise use of automl and the role of humans in the loop. in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.
Dabla-Norris, E. and K. Kochhar, Women, technology, and the future of work. IMF blog, Insights and Analysis on Economics and Finance, 2018.
Truswell, E., As jobs are automated, will men and women be affected equally. Harvard Business Review, 2019.
Karmaker, S.K., et al., Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 2021. 54(8): p. 1-36.
Sandha, S. S., Aggarwal, M., Fedorov, I., & Srivastava, M. (2020, May). Mango: A python library for parallel hyperparameter tuning. In Icassp 2020-2020 ieee international conference on acoustics, speech and signal processing (icassp) (pp. 3987-3991). IEEE.
Sakpere, A. B. (2019, May). Using social platforms to increase engagement in teaching computer programming. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-6).
Yip, W. (2020). Lifecycle of machine learning models 1 2.
Mohr, F., & Wever, M. (2023). Naive automated machine learning. Machine Learning, 112(4), 1131-1170.

No competing interests reported.

AuthorsProfiles.docx

Download PDF

Editorial decision: Revision requested
06 Sep, 2024
Reviews received at journal
05 Sep, 2024
Reviewers agreed at journal
25 Aug, 2024
Reviews received at journal
14 Aug, 2024
Reviewers agreed at journal
03 Aug, 2024
Reviewers invited by journal
03 Aug, 2024
Editor assigned by journal
28 Jul, 2024
Editor invited by journal
19 Jul, 2024
Submission checks completed at journal
10 Jul, 2024
First submitted to journal
08 Jul, 2024

You are reading this latest preprint version

Exploring User Adoption and Experience of AutoML Platforms: Learning Curves, Usability, and Design Considerations

Status:

Version 1

Abstract

Figures

1. Introduction

2. Literature Review

3. Methodology

4. Results and Discussion

4.7 Limitation and Future Work

5. Conclusion

Declarations

Ethical Approval and Consent to Participate

Consent for publication

Availability of supporting data

Completing Interests

References

Additional Declarations

Supplementary Files

Status:

Version 1