The Effectiveness of Automated Speech Recognition in English with Native Thai Speakers

doi:10.21203/rs.3.rs-2125920/v1

Download PDF

Research Article

The Effectiveness of Automated Speech Recognition in English with Native Thai Speakers

https://doi.org/10.21203/rs.3.rs-2125920/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Usability evaluation performance is one of the cornerstones of user interface design. This study presents the usability evaluation of Apple’s Siri based on Nielsen's usability heuristics. Apple’s Siri is one of the technologies of automatic speech recognition (ASR) that allows the user to use a natural language with smartphones and smart devices. The result shows that this feature still has a problem with the user who speaks English as a second language. This study may lead the developer to provide the best voice interfaces for the user who speaks English with a different accent.

ASR

Speech Recognition

Voice Recognition

Siri

mobile devices such as Apple’s Siri. Apple’s Siri plays the role of the intelligent personal assistant that helps the user get things done by voice interaction. The feature uses a natural language user interface (NLUI), which is a type of computer-human interface to answer questions from the user. Although English is spoken in many accents, Apple's Siri currently accommodates only six accents: American, British, Indian, Irish, South African, and Australian. This application still has a problem with the user's accent who speaks English as a second language.

The Apple Siri application has been evaluated for usability in many aspects and has not yet been tested in-depth when it comes to English as a second language (ESL) speakers and broken English. The present ubiquity of the personal smartphone and its increasing necessity for modern life, especially for those lower on the socioeconomic ladder, means Siri’s usability for speakers with less standard English accents. In this usability test, this study focused on Siri’s ability to recognize Thai-accented English and evaluate the solutions it composes for the end user. The continuous progress of programs such as Siri is important for usability, Apple's ability to compete in the market, and both to retain and gain users.

Most software developers require usability testing before the application is launched for the user. The performance of usability evaluation is one of the cornerstones of user interface design [1]. Usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [2]. The previous study has shown that usability evaluation plays a significant role in user interface design. Usability evaluation focuses on how users can learn and use the product to achieve their goals. [1] explained that the choice of an appropriate evaluation methodology is important for how harmful it is if it is applied incorrectly to the actual problem. Many researchers provided evaluation methodologies such as heuristic evaluation [3] and golden rules of interface design [4]. The purpose of this study is to evaluate how well Siri understands English that is not of the main six accents, specifically how well Siri understands English from users who have Thai as a native tongue, which uses heuristic evaluation for measuring usability.

2.1 Speech Recognition Technology

Automatic Speech Recognition (ASR) is the technology that is translated into text between the computer and the user. ASR is an independent computer-driven transcription of spoken language into readable text in real-time [5]. Previous studies show that the ASR has three models consisting of embedded speech recognition, speech recognition in the cloud, and distributed speech recognition. The advantages and disadvantages of three current ASR models on mobile devices were compared and proposed a fourth model of use, which they called shared speech recognition with user-based adaptation [6]. The authors also said that the challenge of assessing the accuracy of ASR in mobile devices affects several factors. For example, limited available storage space, small cache, low processor, and language.

According to [7] presented a versatile post-processing technique based on the phonetic distance that integrates domain knowledge to improve ASR performance. The authors evaluated Google ASR which differs in domain size and show the result of the gap between the superior acoustic modeling performance of cloud-based speech recognition services and relatively weak open-source acoustic models and the high-performance impact of language restrictions provided by domain knowledge.

According to [8] presented, the HomeServices system uses cloud-based automatic speech recognition. The authors adapted personal devices to the homes of people who require voice-controlled assistive technology. They described the idea of a PAL (a Personal Adaptive Listener) and outlined a methodology called “Virtuous Circle”. This study shows that ASR server recognition technology can be built around an internal team of people who use devices as personal assistants.

According to [9] Google Speech was evaluated which was tested in the English language of two Swedish speakers according to the word error rate (WER) and the translation speed. The author compared this with another speech recognition named 'Pocketsphinx', which tests the same recordings and measurements. The results show that Google Speech displayed 28, 5%, 30,27%, 46,1% and 39,94% lower WER scores than Pocketsphinx.

2.2 Usability Evaluation on Mobile

According to [10] evaluated the performance of two popular cloud-based speech recognition applications, Apple’s Siri and Google Speech Recognition (GSR) under various network conditions. The authors provided two models to evaluate the transcription delay and the accuracy of the transcription of each application under different packet loss and jitter values. They said that delay and accuracy of the voice recognition process is an important parameter that affects the quality of user experience with cloud-based speech recognition applications. The result shows that the performance of cloud-based speech recognition systems can be affected by jitter and packet loss, which occur frequently over Wi-Fi and cellular network connections.

According to [11] presented a heuristic evaluation checklist on the mobile interface, which is experimentally evaluated as a design tool. The result shows that mobile usability has different kinds of devices, contexts, tasks, and users. The author divided three types of mobile interfaces. The first type is a feature phone. The authors defined that “they are basic handsets with tiny screens and very limited keypads that are suitable mainly for dialing phone numbers.” The second type is smartphones. The authors defined “phones with midsize screens and full A-Z keypads.” The third type is touch phone/touch tablets. The authors defined “devices with touch-sensitive screens that cover almost the entire front of the phone.” This study shows that the new checklist is useful for avoiding usability gaps in mobile interfaces even with new developers.

According to [12] presented generic heuristics and performed a new theory based on Jakob Nielsen's heuristics. This study derived the initial usability heuristics from Nielsen's heuristics. Generic usability heuristics using a conceptual-theoretical approach suitable for the evaluation of MMA because the set of heuristics by Nielsen is not suitable for MMAs. This study was implemented in four phases. First, they explored the generic heuristics of Nielsen's heuristics, especially evaluating the usability of MMAs. Second, they familiarized themselves with a domain-specific heuristic. Third, they formulated the new heuristics.

3.1 Participants

This study used a test population to represent the user who has Thai as a native language. This study tested Siri's ability to respond to five adult users. The sample of users tested differed in terms of age, experience with Apple’s Siri application, and the number of years speaking English. All users are over 18 years of age. Table 1 presents the user profile in this study. Before beginning the study, the Ethics Review Board (ERB) approval for the testing of human subjects will be obtained.

Table 1

User profile of the participant.
Participants	Age	Gender	Experience with Apple’s Siri
User 1	31	Female	3
User 2	40	Female	0
User 3	50	Male	0
User 4	34	Male	0
User 5	44	Female	4

3.2 Research Instrument

Usability Metrics. This study used qualitative and quantitative data measured by the usability metric. The qualitative data was obtained from the observations of the researcher during the test. For quantitative data based on Hughes's study [13]. Table 1 shows the measurement of quantitative metrics in this study.

Table 2

The quantitative metrics.
No	Measurement
1	The amount of time the user to finish each of the tasks.
2	The amount of time the user to finish the test.
3	The number of errors occurs by each user when Apple’s Siri responds.
4	The number of tasks the user completed in 30 minutes.
5	The number of positive comments from the user.
6	The number of negative comments from the user.

Task List. This study set the task list based on Nielsen's heuristics. The role of the researcher in this study is to act as an observer. The researcher performed the test tasks and observed user interactions with the Apple Siri application. The researcher presented scenarios and tasks on paper in English text, and some may find it difficult to read because this study is being carried out with Thai people who are functionally illiterate in English.

In some cases, the researcher must read the procedure for the user; however, the user was alone holding the iPhone and may ask clarifying questions. The researcher recorded observations on the heuristic data sheet for all the tasks. The user completed the task in 30 minutes. The researcher gave a paper task list to the user and allowed them to ask before beginning the test. Table 3 shows the task list for the evaluation of Apple Siri.

Table 3

Example of Task List for Evaluation of Apple’s Siri.
Number of Steps	Task
Step 1	Go to Setting > Siri > Enable Siri
Step 2	Press and hold the home button to start speaking to Siri. You can say “Hello” or “Hey Siri” for start
Step 4	Speaking to Siri to find your current location
Step 4	Ask Siri to find the restaurant near here Ex. “Is there a restaurant near me?”
Step 5	Follow the conversation about the restaurant. Ex. Ask Siri for directions to go restaurant you want
Step 6	Try to interrupt Siri when you speaking
Step 7	Try to rotate the phone to the other side when you speaking
Step 8	Speaking to Siri to use email Ex. “New email to XXX”
Step 9	Follow the conversation about email until you can send an email to anyone
Step 10	Speaking to Siri to set alarm Ex. Set an alarm for 7:00 am.
Step 11	Speaking to Siri to change alarm Ex. Change my 7:00 am alarm to 9:00 am
Step 12	Speaking to Siri to open the app in iPhone Ex. Hey Siri Take a picture

Test Environment. The testing location was in the living room of the researcher’s home. This space is an open floor plan with hallways to the garage and kitchen and a balcony from the second floor. The smartphone to be used is an iPhone 13 using the operating system iOS 16.

Data Collection. This study measured five users who do not speak English as their native language, specifically Thai speakers. The five users are two men and three women, all over 18 years old. The researcher is bilingual in Thai and English. The component of the study includes five users who must sign the paperwork of the Statement of Informed Consent following the Ethics Review Board (ERB) policy. Each user was tested to follow the task list without any assistance from the researcher. This study does not record video during the test, only observation by the researcher.

This study requires each user to sign the Statement of Informed Consent and a questionnaire to answer the usability test. After the test, the user completed the questionnaire. During the test, each user worked alone. The testing of each user was done for a maximum of 30 minutes. The researcher gave the task list and explained, in the beginning, any questions the user had about Siri. Each user was tested once. The environment of the testing site is no noise from traffic or construction. Not all test subjects have an Apple iPhone, but all users have never used Siri before the test.

4.1 Qualitative Results

The qualitative result was obtained from observations. The researcher directed the task list before the test and during the test, encouraging the user to think aloud. Each user completed the task list in 30 minutes cooperating with the researcher. For some tasks, the researcher requested the user to capture the answer for the compared answer of Siri.

The user came to the test area, and the researcher explained the task list. Then, each user began the test by reading the guideline from the guideline on the testing paper. Next, the user began to speak to Siri following the step of task list. The result shows that Siri does not answer the same conversation with the user even though the user spoke the same question. Given the same command, but Siri understood some user accents (see Fig. 1)

During the test, the researcher observed the individual of the user and put the result in the heuristic evaluation based on the heuristic evaluation [3].

This study used heuristics evaluation. Table 4 shows the result of the heuristics evaluation in this study.

The researchers gave five users on the same command. However, the talk was based on the answer to which Siri responded and followed that answer. Three users who have never used Apple's Siri took the longest time to complete the test. Some command users have to say more than once to give Siri understanding. User 5 took the shortest amount of time to complete the test. The result shows that a good accent and the experience with Apple’s Siri affect the test.

Table 4

Heuristics Evaluation of Apple’s Siri.
Checklist of Heuristics	Evaluation
1. Visibility of system status	Each user can configure the application. They felt that the application was easy to learn
2. Match between system and the real world	The user spoke with Siri which is a natural language and Siri responds to the user as a real conversation.
3. User control and freedom	Siri did not allow the user interrupted when it said. The user had to wait until Siri finished the conversation.
4. Consistency and standards	Some of Siri's conversations are consistent with talking to the user. For example, the user speaks to find a restaurant and then directs to the address. Siri can answer the question with consistency like natural speech.
5. Error prevention	The user cannot find the error that occurred because Siri tries to avoid answers that cannot be answered.
6. Recognition rather than recall	The user is satisfied with Siri’s ability to correct the sentence even when the user speaks the wrong grammar.
7. Flexibility and efficiency of use	The application can support all the other sides even when the user tries to rotate the phone to the other side.
8. Aesthetic and minimalist design	The user could understand and easy to read from the screen of Siri.
9. Help users recognize, diagnose, and recover from errors	The user could not find the help document Siri.
10. Help and documentation	Siri provides the little button on the help menu, but it is not an instruction to use Siri. The user felt difficult when they use the application. Each user takes a long time to learn with Siri.

4.2 Quantitative Results

The quantitative results measured the amount of time, the number of errors, and many tasks the user completed in 30 minutes. Testing of each user is a maximum of 30 minutes. Table 5 shows the time taken to finish the tasks.

Table 5

Time taken to complete the tasks.
Task	User 1	User 2	User 3	User 4	User 5
Time of the user to set Siri (step 1)	0:15	0:30	0:40	0:10	0:20
Time of the user finished the test	20:15	28:00	26:15	18:29	18:45
Total Time	20:30	28:30	26:55	18:39	19:05

After the test, the user completed the questionnaire. Table 6 shows the result of the post-questionnaire of this study.

Table 6

Results of post-questionnaire.
	Strongly Agree	Agree	Undecided	Disagree	Strongly Disagree
1. I found that the application was easy to use	1	2	2	0	0
2. I thought the application was well integrated	3	2	0	0	0
3. I felt impressed with the application	0	5	0	0	0
4. I felt the application is difficult	0	1	1	3	0
5. It takes a lot of time to learn	2	1	0	2	0
6. The application does exactly what I want	1	3	0	1	0
7. This application is frustrating	0	0	1	3	1
8. I found the help documentation	0	0	0	0	5
9. I would use this application frequently	0	1	0	3	1
10. I would recommend this application to another person	1	1	2	1	0

In general, Apple’s Siri can follow user-spoken commands but often does not answer the question directly. The application will go to the web search when it does not know the answer. Apple’s Siri still has a problem with Thai accent. For Apple’s Siri to be truly competitive in the global market, it should recognize as many accents as possible.

Data Availability

Data are available via e-mail at [email protected].

Conflict of Interest

The authors declare no conflicts of interest.

Acknowledgments

I would like thank to all key informants who provided variable time and effort for this study.

Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (Some of the time). Proceedings from the SIGCHI Conference on Human Factor in Computing system ‘08, Florence, Italy.
ISO 9241-11, Ergonomics of human-system interaction – Part11: Usability: Definition and concepts.
Nielsen, J. (1993). Usability engineering. New York: Academic Press.
Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S., & Elmqvist, N. (2016). Designing the User Interface: Strategies for Effective Human-Computer Interaction: Sixth Edition, Pearson.
Stuckless, R. (1994). Developments in real-time speech-to-text communication for people with impaired hearing. In M. Ross(Ed.), Communication access for people with hearing loss (pp.197-226). Baltimore, MD: York Press.
Kumar, A., Tewari, A., Horrigan, S., Kam, M., Metze, F., & Canny, J. (2011). Rethinking speech recognition on mobile devices. In IUI4DR. ACM.
Twiefel, J., Baumann, T., Heinrich, S., & Wermter, S. (2014). Improving Domain-Independent Cloud-Based Speech Recognition with Domain-Dependent Phonetic Post-Processing. Proceeding from the Twenty-Eighth AAAI Conference on Artificial Intelligence (pp.1529-1535) ’14, Quebec, Canada.
Christensen, H. Casanuevo, I., Cunningham, S., Green, P., & Hain, T. (2013). homeService: Voice-enabled assistive technology in the home using cloud-based automatic speech recognition. In 4^th Workshop on Speech and Language Processing for Assistive Technologies (SLPAT) ‘13, Grenoble, France.
Stenman, M. (2015). Automatic speech recognition: An evaluation of Google Speech (Bachelor Thesis) Retrieved from http://www.diva-portal.org/smash/get/diva2:852746/FULLTEXT01.pdf
Assefi, M., Liu, G., Wittie, M. P., & Izurieta, C. (2015). An Experimental Evaluation of Apple Siri and Google Speech Recognition. Retried from: https://www.cs.montana.edu/izurieta/pubs/sede2_2015.pdf
Gomez, R. Y., Caballero, D. C., & Sevillano, J. L. (2014). Heuristic Evaluation on Mobile Interfaces: A New Checklist. The Scientific World Journal, (2014), 1-19. doi: 10.1155/2014/434326
Kuparinen, L., Silvennoinen, J., & Isomäki, H. (2013). Introducing usability heuristics for mobile map applications. In Proceedings of the 26th International Cartographic Conference, Dresden, Germany.
Hughes, M. (1999). Rigor in usability testing. Technical Communication. Journal of the Society for Technical Communication, 46(4), 488-494.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

The Effectiveness of Automated Speech Recognition in English with Native Thai Speakers

Status:

Version 1

Abstract

Figures

1 Introduction

2 Literature Review

3 Materials And Methods

3.1 Participants

3.2 Research Instrument

4 Results And Discussion

4.1 Qualitative Results

4.2 Quantitative Results

5 Conclusion

Declarations

Data Availability

Conflict of Interest

Acknowledgments

References

Additional Declarations

Status:

Version 1