Global events such as pandemics and wars have prompted many individuals to reassess the significance of remote collaboration for training, providing assistance, maintenance, and education. While video conferencing has gained popularity, numerous opportunities still exist for more sophisticated methods of Augmented Reality (AR) remote interaction. Hand-drawn AR sketches/annotations (AAS) are commonly used for expressing visual instructions. However, these freehand drawings are not enough to communicate the specific instructions required for industrial applications. Therefore, oral communication always serves as a critical supplement for addressing misinterpretations and language barriers. In such cases, our work is dedicated to sharing clear instructions based on AAS by the adaptive transformation of instructions (ATI) method. As a result, we present a Spatial AR(SAR) remote collaborative platform that can support converting AAS into standard symbols which provides clear guidance and has been widely accepted by the public. We conduct a formal user study to evaluate the prototype platform concerning performance time, general collaborative experience, usability based on ranking, and users’ preferences. The results indicated that ATI-based cues have a positive rule on remote collaborative training tasks in terms of user experience. More significantly, our work provides valuable implications on the way for further study of gesture-based interaction in AR remote collaboration on training tasks.
Research Article
ATI-driven SAR Remote Collaboration Platform for Manufacturing in a Training Task
https://doi.org/10.21203/rs.3.rs-3618053/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Adaptive Transformation of Instructions (ATI)
Spatial Augmented Reality (SAR)
Augmented Reality (AR)
Remote Collaboration
AR annotations (AAS)
In this paper, we center our efforts on exploring the potential influence of Spatial Augmented Reality (SAR) remote collaboration on a training task through the application of Adaptive Transformation of Instructions (ATI) about work efficiency and user experience. The travel cost and time, the COVID-19 pandemic, and the desire to reduce carbon emissions made remote collaborative work increasingly acceptable, if not essential[1–6]. Specifically, these factors distinctly promoted the popularity of remote collaborative activities, as is the case of training, emergency response, and medical and industrial domains[7, 8]. Although video-conferencing has prospered, Augmented Reality (AR) technology provides lots of opportunities lie ahead for more advanced strategies for remote collaboration platforms in training/ procedural tasks and remote assistance [1, 3, 9–12]. For successful AR-based remote collaboration, it is key for partners to timely proper instructions[13], and it can improve user experience and provide a platform that enables two or more partners geographically separated from one another to feel like they are virtually co-located work[13–15]. To ensure business sustainability and positively embrace Industry 4.0, most enterprises hope that workers can be effectively trained making use of the advantages and the capabilities of AR remote collaboration[8, 16, 17]. More importantly, the remote collaborative experience is essential to elevate the output and quality of the products, and there is still a lot of tasks that need to be done manually by workers in manufacturing[18].
Taking full advantage of AR and remote collaboration, we discussed how AR annotations (AAS), which can create common ground for collaborators[14, 19], based on verbal cues to share facilities and practical information remotely and improve remote collaborative training tasks in manufacturing. Previous studies have investigated which various non-verbal cues (e.g. AAS, eye gaze, gesture, 3D virtual replicas) are positively effective for instructions sharing and understanding a remote collaborative context, such as 3DGAM[16], EaVAS[20], 3DSAN[21], ARA-based cross‑platform[22], 2.5DHANDS[23], BeHere[15] and TeleAdvisor[24, 25].
There has been little research that explores converting AAS cues into standard symbols for AR-based remote collaboration on training in manufacturing[16]. There is some research on interaction and visualization methods based on AAS in VR/AR remote collaboration on training, maintenance, and assembly[3, 4, 8, 14, 21, 26–29], as shown in Fig. 1. To improve remote maintenance efficiency, Scurati et al. [26] investigated how to create and adopt an exhaustive and standard vocabulary of symbols representing AR instructions. Specifically, they first found the common maintenance actions in manuals based on two constraints of conflict set and reversibility and attempted to convert them to standard symbols on the basis of the criteria of guessability and homogeneity. Then, a case study of AR remote maintenance was conducted for the effectiveness of the prototype system. Wang et al.[14] developed VR/SAR remote collaborative platform sharing AAS using haptic feedback with tangible interaction, and discovered AAS cues and haptic feedback are beneficial to remote collaborative tasks, in particular, the remote users’ experience. Zhang et al.[21, 27] studied how 3D sketch and spatial area cues affect remote collaborative assembly and these visual cues can improve assembly efficiency and user experience, and reduce operation errors. The research lays the cornerstone of using AAS in VR/AR-based remote collaborative tasks. Furthermore, the literature review has proved the advantage of providing real-time assistance using AAS, which is very popular in AR remote/local collaboration, because the AAS-based approach is more flexible than others in terms of efficient deictic references and a freestyle drawing[9, 11]. However, existing solutions for providing assistance using AAS require remote experts to perform real-time task demonstrations, even multiple times, especially in training tasks.
Therefore, there is an opportunity to study how the ATI-driven SAR remote collaboration platform influences training tasks in manufacturing. More importantly, the repaid development of artificial intelligence technology will open unlimited potential for the realization of ATI in remote collaboration. To the best of our knowledge, there was no prior research exploring ATI based on AAS taking full advantage of the AAS and SAR interface in training tasks. Our work provides three-fold innovative and remarkable contributions:
-
We design a novel SAR remote collaborative platform, ATI, making use of SAR in the local site, providing instructions based on AAS, whilst sharing ATI cues based on verbal cues.
-
The first user study is to explore how sharing ATI cues affects SAR remote collaboration on training tasks by comparing two interfaces: the popular approach currently AAS is the baseline solution and ATI-based cues.
-
Discussion of the benefits and implementation details of utilizing ATI-based visual cues to enhance SAR remote collaboration during training tasks.
The structure of the paper is as follows. we first review related works. Second, we describe the implementation details of our proposed ATI. Then, we explain the details of the formal user study to evaluate the prototype system and present the discussion. Next, limitations and open issues are presented. Finally, some conclusions are given in the last section.
AR remote collaboration for supporting assistance or training, on-site workers always need to operate specific devices(e.g., replacement or disassembling parts) when have little relevant experience or knowledge to efficiently perform the task, in such cases, it is very critical to enable remote experts can provide local workers with non-verbal cues on the no-site. This includes two significant challenges, that is, how to enable remote experts to easily and intuitively express cues, and how to convert AAS-based cues into standard guidance symbols. In this part, we will review related works on AR remote collaborative work in three points around these two critical challenges.
2.1 SAR-based remote collaboration
According to survey papers[1, 11, 12, 30, 31] closely related to AR remote collaboration, the SAR interface is one of the most popular conditions(e.g., desktop-based, handheld, head-mounted displays(HMD), SAR-based interface) used on the working site and the on-site workers, significantly, prefer the SAR-based interface because of freeing up hands and no need to wear a helmet. As confirmed by the literature, there is much research on SAR-based remote collaboration in training, repair, assembly, etc.
Baumeister et al.[32] investigated into the influence of AR instructions in terms of users’ cognitive load using three interfaces: SAR, AR-HMD, and VR-HMD, and found that the SAR interface has the advantages of reducing the word and cognitive load. Recently, Wang et al.[33] developed SAR-based assembly guidance platform, UcAI, converting the expert’s potential experience into AR-based instructions of effective visual features. The control user study showed that UcAI has a remarkably positive influence on reducing users’ cognitive load, improving assembly efficiency, and reducing the potential for confusion or error, Specifically, in high-precision operation tasks. Due to the rapid development of gesture recognition sensors, gesture-based cues provide a new world for AR remote collaboration. Wang et al.[23] discussed the effects of sharing the 2.5D gestures in a SAR remote collaborative assembly compared with the AAS condition. They found that gesture-based cues and AAS both have a significant impact on remote collaborative tasks concerning the average completion time and user general collaborative experience. Higuchi et al.[34] designed a SAR remote collaborative system based on sharing gaze and gesture cues, which can real-timely guide the local user to build a Lego setting task. Nevertheless, this SAR remote collaborative platform has a limitation of the fractured ecology[15]. In particular, when the remote expert interacts with the mobile device, the no-site users may confuse the spatial relationship between hands and virtual objects. To overcome this limitation, Wang et al.[35] evaluated the gesture and head pointing (GHP) cues in SAR remote collaboration. They believe that the SAR non-verbal cues in a typical assembly task can improve the collaborative experience. Furthermore, Wang et al.[36, 37] proposed an AR remote collaborative platform for training in manufacturing employing 3D virtual replicas, and found that AR instructions based on the combination of gestures and virtual replicas have a positive influence on user experience, such as presence, attention, enjoyment, and confidence.
Although there is some MR remote collaborative research on training, assembly, and maintenance, they focus on the advantages of the naturalness and flexibility of non-verbal cues based on AAS, gestures, and 3D virtual replicas. More significantly, a large body of study points to the fact that SAR visual cues have quite a positive effect on remote collaboration on performance and general user experience.
2.2 Sharing AAS cues
AAS-based cues are widely used in remote collaboration on assembly and training because it’s easily implemented and freely and naturally express the key visual communication information[1, 11, 12, 30, 38]. There has been a large number of scholars have been putting a premium on how remote experts naturally and intuitively express AAS and improve remote collaboration platforms using AAS cues.
A versatile SAR remote collaboration system, TeleAdvisor, was designed by Gurevich et al. [24, 25] based on the pair of camera and projector in the local site to connect the remote expert using a desktop-based interface creating AAS for performing a Lego construction task and equipment maintenance. They found that TeleAdvisor is a hands-free, practical, and low-cost strategy that improves remote collaboration in terms of task efficiency and understanding, in particular, minimizing the local users’ cognitive load. Recently, Wang et al.[23] proposed a SAR remote collaboration platform providing AAS and gesture cues, and the results showed AAS cues are more beneficial to remote users naturally expressing instructions. Kim et al.[39] developed an AR remote collaborative platform to explore the influence of three different visual cues (e.g., gestures, pointer, AAS), and the results showed that AAS, particularly, the combination of gestures and AAS, requires less cognitive effort, provides better communication cues, and increases the feeling of co-presence in the case of the dependent view. Wang et al.[14] also evaluated VR/SAR remote collaborative platform sharing AAS using gestures, and discovered AAS cues are beneficial to remote collaboration on building a Lego setting task.
The rapid development of AR technology and the deployment of advanced processors in mobile devices have opened up novel opportunities for improving AR remote collaboration. Latterly, Marques et al. [3, 4, 7, 8, 28, 40, 41] systematically studied how the AAS cues impact AR remote collaboration by taking advantage of mobile devices (e.g., smartphones and tablets) in local sites, as shown in Fig. 2. This AR remote collaboration system architecture adopts the desktop interface in the remote site based on a laptop and the AR interface in the local site based on a mobile phone, providing domain experts’ AAS instructions for local workers in maintenance and assembly tasks. They found a list of key requirements guiding developers in creating AAS cues, the stabilized AAS and video efficiently providing important ways for remote experts to cooperate with local professionals, and the critical strategies procedure on comprehensively understanding the remote collaborative process. There are lots of researchers who have explored several ways to provide a better awareness based on AAS. Cidota et al. [42] developed an AR virtual colocation system sharing visual and verbal cues, and results showed that, compared to verbal communication, the visual cues were significantly preferred by users over verbal or no notifications. Then, Kim et al.[43, 44] extended Cidota’s work to investigate how remote users can share ideas with local users by AAS-based cues, providing peripheral visual communication cues, to perform a real-world task under the condition of having an independent or a dependent view.
Most center attention on easily and intuitively expressing cues and the use of AAS to augment the on-site setting, such as sketches, pointers, and pre-designed visual cues (e.g., arrows), effectively enhancing remote collaborative tasks[8, 45, 46]. In daily life, sketches or annotations are a key interaction way always being used to highlight and indicate certain real-world objects of collaborative environment or to add signal tags and explanations. Importantly, the combination of sketches or annotations and AR remote collaboration technology can provide a powerful method of efficiently offering partners more visual cues facilitating overall mutual understanding of the working settings. It should be noted that the above-mentioned works showed that sharing AAS cues is very useful for meliorating remote collaborative tasks, and the AAS cues are one of the important visual cues commonly used in remote collaboration. Thus, we firmly believe that sharing AAS cues also drives performing the training task at an industrial manufacturing center.
2.3 Converting visual cues
During manual operations, reasonable AR-based guidance instructions are essential for improving performance efficiency and psychological aggregation, and minimizing cognitive and psychological fatigue[33, 47]. This view can be proved in user behavior. When the AR-based visual cues meet the user’s need recognition, workers require less experience and relevant information retrieval times and workload efforts to know the collaborative intention[33].
The issue of enhancing visual guidance cues for AR instructions has gained attention in recent years. Mohr et al.[48] proposed a prototype system that can convert the key guidance procedure in the printed technical documentation into three-dimensional AR animations registered to the physical object by recognizing annotated explosion diagrams. The system can generate an interactive AR presentation which can be useful for assembly training and understanding the structure of products. Engelke et al.[49] presented a new concept being enable experts to convert task descriptions into AR-based instructions, and the results showed the efficiency of the concept and the application.
For an assembly training task in the industrial sector, Wang et al.[36] proposed an adaptive MR remote collaborative platform, reducing user demonstration operations, that the remote expert can easily interact with three-dimensional virtual replicas on a Virtual Reality (VR) site, then the guidance animation can be displayed in partners’ views, allowing the local worker to assemble the product following these instructions. For on-routine maintenance, Kritzler et al.[50] designed a providing remote assistance system, RemoteBob, that could enable remote experts to share guidance instructions using AAS and virtual replicas with on-site workers, and found that virtual replica cues could avoid miscommunication, provide clear visual cues, and reduce operations errors. However, there is a fractured ecology of the local or remote interface which easily makes a distraction for users. For remote maintenance, Scurati et al. [26] introduced an exhaustive and standard vocabulary of symbols representing AR instructions on the basis of guessability and homogeneity, and they believe that the proposed approach is a starting point for determining criteria principles, accepted by AR manuals, for designing AR instructions.
The aforementioned study shows that AR instruction design should meet users’ mental representation and take into account users’ cognitive state as well as common sense[51, 52]. Recently, although AI-based VR/AR interfaces increasingly attract more and more attention by using deep learning in training that is relevant for maintenance and assembly, there is little research keeping an eye on ATI in remote collaboration on training. Thus, for assembly training in an industrial manufacturing center, these researches have room to be improved by ATI of visual guidance cues, particularly, AAS.
2.4 Summary
Based on the comprehensive review of the aforementioned research, we can extract three significant insights. First, with the rapid development of VR/AR technologies and suitable VR/AR devices becoming available, an increasing number of visual communication cues (e.g., AAS, gestures, gaze) have been exploited to enhance remote collaboration for assembly training and maintenance. Nevertheless, the research did not focus on converting visual communication cues into standard symbols in AR, as a result, there was little research on ATI. Second, although lots of researchers explored sharing AAS cues in SAR remote collaboration for physical tasks, they did not consider how to transfer AAS cues to standard AR instructions in industry and explore the effect of ATI in remote collaborative tasks. Third, providing assistance based on AAS is a good way to enhance AR remote collaboration concerning performance. However, the guidance cues based on AAS have a certain degree of arbitrariness when created by remote experts, which can darkly affect collaboration efficiency, leading to more verbal cues for better understanding to each other.
Therefore, to address the limitations of prior research on sharing AAS cues and open the potential of AAS in remote collaboration on training, we designed a novel SAR remote collaborative platform. Firstly, the platform can share the AAS-based visual cues which are naturally and intuitively created by remote experts to express instructions and improve mutual collaboration using a computer that is the most widely used and familiar interaction interface in our daily life and work. Secondly, the platform combines ATI cues and AAS notifications in a complementary way to allow remote users to designate the specific area or objects in SAR on-site working settings, and to better understand and grasp the dynamic interplay between collaborative members influenced by cultural differences and personal experiences and more. Thirdly, using the platform, we conducted a formal user study in which local workers followed the instructions provided by remote experts to perform an assembly training task in a SAR workspace by using two conditions of AAS and ATI visual notification.
It is important to note that the specific needs and preferences of remote collaborators may vary depending on the nature of their work and the tasks they are performing. As such, it may be useful to conduct further research or gather feedback from local/remote collaborators to better understand their needs and preferences when it comes to SAR remote collaboration based on sharing AAS and interfaces. Through these efforts, we hope that a better understanding can be gained of the interaction situation and mutual collaboration between local workers, enabling the development of more effective strategies and measures of offering better performance and a more immersive experience for training tasks.
In this section, we elaborate on the platform and implementation details from three parts: (1) system architecture, (2) creating AAS, and (3) ATI.
3.1 System architecture
According to review papers[1, 11, 12, 29–31] on VR/AR remote collaboration, the PC interaction interface (desktop and laptop) is often adopted by remote collaborators because of their stationary and ability to provide adequate computing power[31]. This may be especially important for collaborative tasks that require guiding or assisting a local user in real time. Recently, Marques’ research[3, 4, 7, 8, 28, 40, 53] has confirmed this. Additionally, the SAR-based interface is a versatile and convenient option for on-site users[10–12] which has several advantages over other types of interfaces(e.g., HMD, mobile phones, and tablets), such as being hands-free, not requiring the user to wear an HMD, and allowing the user to interact more naturally and intuitively. This makes them particularly suitable for use in situations where users need to have their hands free for other tasks, such as in medical and industrial applications. And, Wang's research[14, 23, 35, 54, 55] has also confirmed this.
So, our platform includes three main modules(see Fig. 3): (1) a local SAR side aids the on-site workforce in receiving and complying with AAS/ATI to accomplish tasks, (2) a remote side aids the remote expert to create AAS/ATI cues and observe the state of performing tasks, (3) a server connects the local SAR client and the remote expert client and is responsible for synchronizing data.
The prototype platform, running on Windows 10 OS, was developed by Unity 3D1 2018.3.8, Wampserver2, Logitech camera SDK, and OpenCvSharp3. We employed an Intel® NUC as a server to link the remote side client and the local SAR side, client, via a Wi-Fi 5 wireless, and the network was constructed upon the TCP/IP protocol. To simplify the platform, the prototype system did not have an audio channel. However, local/remote participants could speak to each other using WeChat4. By eliminating unnecessary complexities, we can concentrate on the essential elements that matter the most for exploring the effect of AAS/ATI in remote collaboration. For the local SAR side, the client system was running on an HP laptop 14-ey0031AU (CPU AMD R7, RAM 32G, SSD 1T), an XGIMI projector, and a Logitech camera. The on-site users can see the AAS/ATI cues which are projected on the real scene. And the Logitech camera can real-timely capture of whole on-site settings meanwhile the remote expert can have a real-time or near-real-time view of what the local user is seeing by the shared video streaming. This allows the remote expert to provide guidance, instructions, or feedback based on the visual information provided by the RGB frames. For example, in fields like remote technical support, the remote expert can visually assess a situation, identify issues, and provide appropriate recommendations. This shared visual understanding can enhance communication and reduce misunderstandings between the two parties. It also allows for efficient problem-solving and decision-making processes, as the remote expert can directly observe and analyze the visual details of the scene. In addition, for the remote site, the system was built upon an HP desktop computer (CPU Intel i7-10700K, RAM 32GB, SSD 2T). The remote experts can naturally and easily create AAS by the PC interaction interface.
3.2 Creating AAS
The proposed SAR remote collaboration system can support the creation of AAS on the shared video streams using a mouse in the remote site. This feature is a key part of the system, as it has the potential to provide a highly effective and intuitive way for remotely located users to quickly and easily point out and annotate specific details, highlight areas of interest, or otherwise engage with specific areas or features of the no-site working state for further discussion or consideration. This level of interactivity and engagement can significantly enhance communication and understanding between remote collaborators.
The algorithm framework of AAS is illustrated in Fig. 5, which mainly consists of four steps. First, it communicates with the server to receive the shared video stream that was rendered in a game object of Unity 3D. Second, it obtains the coordinate points and key points to create sketches using linerender, based on removing noise through Kilman filtering. If the coordinates of mouse movement are Ei (xi, yi, zi), the extracted key point coordinates are ei (ui, vi). The original real-time key point fluctuations are large, and it is necessary to use Kalman filtering to reduce noise and increase data smoothness and robustness. This can be expressed as:
\({\mathop e\limits^{ \wedge } _k}={\mathop e\limits^{ \wedge } _{k - 1}}+{g_k}({z_k} - {\mathop e\limits^{ \wedge } _{k - 1}})\)
where, \({\mathop e\limits^{ \wedge } _k}\) is the estimated current coordinate of the mouse, \({\mathop e\limits^{ \wedge } _{k - 1}}\)is the estimated position of the previous mouse movement, \({g_k}\)stands for the Kalman gain, and \({z_k}\)is the current position of the mouse.
\({g_k}={p_{k - 1}}({p_{k - 1}}+r)\)
\({p_k}={p_{k - 1}}(1-{g_k})\)
where r is the variance of the noise which is set as a constant in the calculation, \({p_k}\)is the covariance between the position estimation value and the mouse movement coordinate at time k in the recursive calculation
Third, the user can create fine-grained sketches by activating a button on the interface or by long-pressing the left mouse button, and these sketches can be removed by activating the 'clear' button or by clicking the right mouse button. Finally, the sketches are shared with the local SAR site, and AAS is superimposed on the no-site settings to provide real-time instructions.
3.3 ATI
Prior many researches[3, 4, 7, 8, 11, 14, 21, 24–26, 28, 40] have demonstrated that AAS cues are important for AR remote collaboration on physical tasks, and more importantly, we discovered that their results showed that the most frequently used AAS cues have two types, such as circles/ellipse, arrows, as shown in Fig. 6. We used OpenCvSharp, Vectrosity5.65 to create AAS and convert it into ATI cues, as shown in Fig. 4 and Fig. 7. Generally speaking, the algorithm encompasses four critical steps: (1) Images acquisition. In the remote client system, AAS cues were created on a new layer that is prepared for sketching. When the virtual camera captures images of AAS in the unity 3D scene, other objects and disturbing elements can be excluded, laying a solid foundation for subsequent image processing. (2) Recognition of hand-drawn sketch features. We employed grayscale conversion, filtering, morphological operations, and a Line Segment Detector(LSD) to detect lines, and then extract their endpoints. (3) Symbols conversion. For circles or ellipses, the outlier points were eliminated by random sample consensus (RANSAC) [56], followed by fitting ellipses or circles using Vectrosity’s FitEllipse function. For arrow cues, a straight line was first fitted through Least Squares, and the points obtained defined the start and endpoints. Secondly, the number of points near two endpoints of the straight line was determined, then defining the side of the line on which the arrow is positioned. (4) the AAS and ATI visual cues would be real-time shared with the on-site and projected onto the real workspace.
In the research, we were interested in exploring the effects of different visual cues (e.g., AAS and ATI) for a training task. A formal user study was conducted for this aim, and we provide a detailed description of experiment and results.
4.1 Experiment
4.1.1 Conditions
Our main independent variable was the different types of visual cues, for instance, AAS and ATI. Thus, we set two conditions and compared their effects on SAR remote collaborative training tasks requiring visual guidance information. The two conditions in the user study are as follows, and it should be noted that verbal communication is available in two conditions.
-
AAS: The platform can provide visual instructions for AAS. So, the remote expert can use the PC interaction interface to naturally and easily create sketches, which is a commonly used way in AR remote collaboration.
-
ATI: In this case, the platform can convert AAS into standard symbols. In addition to the AAS cues, the remote export can guide the local users using ATI. In the default situation, AAS would automatically convert into ATI, and the user can activate the ability to create AAS when the sketch is complex.
5.1.2 Training Task and Hypotheses
For the task content, we chose a typical training task as illustrated in Fig. 4. Additionally, we have two constraints for encouraging collaboration: (1) the local SAR user should take only one part or tool at a time for performing tasks., and (2) the local user must perform the training step by step based on the visual cues provided by the remote expert.
Our study concentrates on the evaluation of the impact of AAS and ATI cues separately for a training task. Consequently, we mainly pay attention to the following two questions: (1) What are the benefits of converting AAS cues into ATI cues for a training task in SAR remote collaboration? (2) How does the sharing of ATI cues from the remote site affect SAR remote collaboration on a training task?
Generally, the two visual cues (e.g., AAS, and ATI) have different advantages in providing visual instructions. Additionally
AAS cues has been proven that have a positive impact to a certain degree on SAR remote collaborative task concerning user experience, usability, and efficiency. Nevertheless, no prior research further explored the conversion of AAS cues. Therefore, four hypotheses are listed below:
-
H1: The ATI condition would provide faster performance than the AAS condition.
-
H2: Compared to the AAS condition, the ATI condition will provide a better general collaborative experience.
-
H3: The ATI condition will offer a superior level of usability based on ranking compared to the AAS condition.
-
H4: Users will prefer the ATI condition over the AAS condition.
4.1.3 Participants
In the research, we recruited 24 participants, including 15 males and 9 females, aged 19–30 (Mean 24.36, Standard Deviation 1.98). All participants are undergraduates and graduates, and their major backgrounds varied, including Intelligent Manufacturing Engineering, Robotics Engineering, Mechanical Engineering, and Industrial
design. All participants voluntarily joined this study after being fully informed of the academic ethics, individual privacy, and safety issues involved in the experiment. We presented the participants’ background on related technologies in the study, as shown in Fig. 8. It shows that most students are novices in SAR remote collaboration, alleviating the influence of the students’ professional backgrounds on the research.
4.1.4 Procedure
The experiment procedure consisted of six steps as shown in Fig. 9. Each trail approximately lasts 42 minutes without interruption. However, we will take a break if the participants wish to rest. All participants were randomly allocated to either remote experts or local staff, and they kept the same roles throughout the experiment by within-subject design. Firstly, we took approximately 3 minutes to introduce the experiment and answer questions to all participants. Secondly, our team collected all participants’ professional information, such as age, gender, major, and related experience. Thirdly, training. Each participant needs to be familiar with the operation of the interfaces and can smoothly and fluently perform the remote collaborative tasks. Subsequently, performing an experiment and filling questionnaires. Specifically, the performance time was recorded, and the user collaborative experience questionnaire, a 7-point Likert scale, which we designed and adapted from Wang et al’ literature[16]. The general user experience questionnaire contains 8 questions for all participants and two additional questions for local and remote participants, respectively. In addition, we used the System Usability Scale (SUS) to evaluate the usability of our platform, as others have done[13, 16]. Last, we conducted brief interviews to gather insight into participants' perspectives on remote collaboration while using each condition.
After finishing the experiment, we asked all participants to rank two interfaces that they had experienced from worst to best based on a preference questionnaire according to Marques et al.[7]. And we focus on seven items, such as, R1 helpful, R2 engaging, R3 novel, R4 familiar, R5 easy-to-use, R6 flexible, R7, customizable.
4.2 Results
In this section, we presented the analysis results. We measured the quality of our platform using both objective measures such as task performance, and subjective measures such as SUS, general collaborative experience, and preference rank.
4.2.1 Performance
Before proceeding with the analysis, we validated the consistency and normality of the data through the Shapiro-Wilk test. The results showed that the data of AAS (p = 0.547) and ATI(p = 0.546) have no deviation. Then we conducted a paired t-test (α = 0.05), and the results did not show a significant difference [t(11)=-1.382, p = 0.194] between the two conditions. The average completion time is 462.25 ± 29.65 s and 470.08 ± 29.02 s for the AAS interface and the ATI interface, respectively, as shown in Fig. 10.
4.2.2 SUS
We evaluated the prototype platform by the SUS and reported the evaluation results as shown in Table 1 and Fig. 11. For a more comprehensive understanding of how to interpret and calculate SUS scores, readers can consult the official SUS website6. In the remote site, the average SUS scores were 73.96 and 75.21 for the AAS interface and the ATI interface, respectively. Additionally, the scores were 77.92 and 78.96 for the AAS interface and the ATI interface, respectively. More importantly, the results did not show a significant difference using Wilcoxon’s signed-rank test (α = 0.05) between the AAS and ATI interfaces. Figure 11 illustrates, overall, the average of SUS in the local SAR site is better than the remote site.
| Interfaces | Clients | M | SD |
|---|---|---|---|
| AAS | Local sites | 77.92 | 3.96 |
| Remote sites | 73.96 | 3.61 | |
| ATI | Local sites | 78.96 | 3.60 |
| Remote sites | 75.21 | 4.05 |
4.2.3 General collaborative experience
About the general collaborative experience based on Table 2, the results showed that the different interfaces had a significant effect with respect to both local and remote sites, as illustrated in Fig. 12 and Fig. 13. Wilcoxon Signed-Rank test (α = 0.05) was performed for each questionnaire to study whether there was a significant effect.
For remote sites as shown in Table 3 and Fig. 12, participants’ feedback indicated that they were more focused (QS4 ), enjoyment(QS5), and confidence (QS6) about providing instructions in real-time when they used the ATI interface (QS4: Z = -2.598, p = 0.009; QS5: Z = -2.668, p = 0.008; QS6: Z = -3.134, p = 0.002 ). In particular, participants felt that by using the ATI interface they could provide instructions more easily than the AAS interface. Besides, the results did not show a significant difference on questions QS1, QS2, QS3, QS7, QS10.
| QS# | Local site | Remote site | |||
|---|---|---|---|---|---|
| Z | p | Z | p | ||
| QS1 | notice | -2.521 | 0.012 | -1.043 | 0.297 |
| QS2 | presence | -0.061 | 0.951 | -0.632 | 0.527 |
| QS3 | attention | -2.676 | 0.007 | -0.707 | 0.480 |
| QS4 | focus | -0.513 | 0.608 | -2.598 | 0.009 |
| QS5 | enjoyment | -2.850 | 0.004 | -2.668 | 0.008 |
| QS6 | confidence | -2.709 | 0.007 | -3.134 | 0.002 |
| QS7 | reciprocation | -0.187 | 0.851 | -0.378 | 0.705 |
| QS8 | working together | -1.134 | 0.257 | -0.707 | 0.480 |
| QS9 | understanding | -3.115 | 0.002 | ||
| QS10 | usefulness | -0.277 | 0.782 | ||
| QS11 | providing instructions | -2.976 | 0.003 | ||
| QS12 | assistance | -1.311 | 0.190 | ||
For the local site as shown in Table 2 and Fig. 13, participants’ feedback indicated that the ATI interface can enhance users' experience by providing additional notifications (QS1 Z = -2.521, p = 0.012), attention (QS3 Z = -2.676, p = 0.007), enjoyment (QS5 Z = -2.850, p = 0.004), and confidence (QS6 Z = -2.709, p = 0.007), making the users more engaged in the tasks. In particular, local participants felt that using the ATI interface can provide information that is easier for us to understand(QS9 Z = -3.115, p = 0.002). Besides, the results did not show a significant difference on questions QS2, QS4, QS7, QS7, QS8, QS10.
Table 3 The general collaborative experience
4.2.4 Ranking and Preference
Figure 14 illustrates the ranking results (1 = the best, 2 = the worst). For all participants, the ATI interface generally ranked higher than the AAS condition. To analyze if participants ranked We conducted a chi-square test ((α = 0.05) to analyze whether there were significant effects. For the remote site, the results showed that there was a significant difference on questions of R2(engaging, χ2(1) = 10.222, p = 0.001), R3(novel, χ2(1) = 5.750, p = 0.016), R4(familiar, χ2(1) = 5.750, p = 0.016), R6(flexible, χ2(1) = 5.750, p = 0.016), except for R1, R5, and R7(p > 0.4)
For the local SAR site, the results showed that there was a significant difference on questions of R1(helpful, χ2(1) = 10.222, p = 0.001), R2(engaging, χ2(1) = 5.914, p = 0.015), R3(novel) and R4(familiar)(p = 0.016), except for R1, R5, and R7(p > 0.4).
When it comes to users’ preferences, all users selected the interface they most preferred. The results showed that most participants like the ATI interface as shown in Fig. 15. Specifically, almost 71% (8 remote + 9 local users) of them chose the ATI interface, and 29% (4 remote + 3 local users) of them chose the AAS interface. In General, most users liked the ATI interface.
In this section, we assessed the effectiveness of sharing ATI on SAR remote collaboration in a training task, especially for training tasks requiring remote guidance. We developed two different interfaces (e.g., AAS and ATI) and assessed them in terms of four hypotheses, such as performance time(H1), general collaborative experience(H2), usability(H3), and preference(H4), by collaborative training tasks. Generally, the results indicated that there is a significantly better quality of remote collaborative experience by converting AAS into standard symbols from subjective perspectives.
For performance time, in the formal user study, the results showed that there wasn’t a significant effect on performance time between the AAS interface and the ATI interface (see Fig. 10). Thus, our hypothesis H1 didn’t support to a certain extent. From the results, we found that the AAS interface is faster than the ATI interface on the average time to complete training tasks. This could be because the AAS interface can provide useful instructions for completing tasks using the commonly used way in our daily life with the assistance of verbal communication. Although the ATI interface can provide clear instructions for the local participants leading to reducing verbal communication, the advantages can be proved from the participant's interview.
As for SUS, the average scores are bigger than 73(see Fig. 10 and Table 1), which shows the developed two systems are well. Wilcoxon’s signed-rank test conducted did not indicate there was a significant difference between the ATI interface and the AAS interface. The results show that the usability of both systems is sufficient to support further formal user study for exploring the effect of converting AAS into standard symbols.
Regarding general collaborative experience, remote participants using the ATI interface were more focused, engaged, and confident compared to those using the AAS interface. This is because the ATI interface provided clear real-time instructions based on industrial training tasks, supporting converting potentially ambiguous or arbitrary annotations into standard symbols. This conversion process was beneficial for remote participants providing instructions. Specifically, remote participants using the ATI interface were more focused, engaged and confident in performing the tasks. Additionally, the ATI interface provided a significantly stronger feeling of enjoyment and confidence compared to the AAS interface. With regard to attention, the ATI interface also provided more cognitive and attentive feedback. Overall, there were significant differences between local and remote participants during remote collaborative tasks, as shown in Fig. 12 and Fig. 13. Therefore, hypothesis H2 (general collaborative experience) is confirmed.
Concerning usability based on users’ ranking, we also evaluated the pretotype platform according to ranking questionnaires with respect to seven perspectives, R1 helpful, R2 engaging, R3 novel, R4 familiar, R5 easy-to-use, R6 flexible, R7, customizable. Specifically, the ATI interface in general better than the AAS interface for both local and remote participants. The ATI interface has several outstanding characteristics that are engagement, novelty, and familiarity for all participants in both local and remote sites. Additionally, the ATI interface can support flexibility and help for the remote site and local SAR site, respectively. A chi-square test was used and results showed that there was a significant effect on usability on the basis of users’ ranking as shown in Fig. 14. Thus, hypothesis H3(usability) is supported to some extent. More importantly, from the interview, some local participants gave positive comments such as “When I progressed and paid attention to training tasks, I didn’t notice my partners’ absence; The visual ATI cues, which are superimposed on the actual working environment, can liberate our hands and enhance our focus, enabling us to enjoy the tasks more; The ATI interface can offer clear instructions, thereby reducing the need for verbal communication. The system would be ideal if it allowed us to create ATI cues.”. Furthermore, some participants in the remote site said: “It's incredible to watch my hand-drawn sketches transform into perfect circles or ellipses; It would be fantastic if the ATI interface could support the conversion of more hand-drawn sketches into standardized symbols.”.
With respect to users’ preferences, we found a clear preference for the ATI interface among both local and remote participants. Although the completion time did not show a significant difference, most users still like the ATI interface. These findings may have resulted from the feedback of the recruited participants in the face-to-face interview. For instance, some participants stated that the standard symbols used in the ATI interface are commonly found in engineering drawings and manuals, making them easy to comprehend. Thus, the hypothesis H4 is accepted.
It should be noted that in our designed platform, the integration of SAR into the onsite settings of the remote collaboration system adds layer of functionality and value. The ability to transmit instructions or visual cues from a remote site to the collaborators allows for informed decision-making and efficient problem-solving. This technology can be applied across a range of fields, including disaster response, security monitoring, urban planning, and more. Overall, the proposed SAR remote collaboration system can provide visual cues (e.g., AAS, ATI) which is a very promising and useful tool for enhancing communication, improving decision-making, and promoting efficient collaboration between remotely located users.
A discussion on the benefits and implications of utilizing ATI and visual notification to enhance MR remote collaboration can be formatted as follows. First, one benefit of using the ATI cue is that it can help improve understanding and communication between remote collaborators. The ability to visualize task instructions allows remote collaborators to quickly identify key elements and areas within a given context. Additionally, this approach can provide a more immersive experience that encourages participants to become more actively involved in the collaboration process. Moreover, visual notifications provide an effective means of communicating important information related to interactions between remote collaborators and annotated areas or objects. Visual notifications allow for a timely update of relevant information, enabling remote collaborators to quickly identify potential issues or opportunities within the collaboration environment. This aspect can significantly improve efficiency and coordination between collaborators, leading to more productive outcomes.
About broader implications, the use of ATI cues and visual notifications may contribute to the development of more advanced collaborative technologies and methods. The integration of such features into future collaboration platforms has the potential to support more seamless and intuitive collaboration experiences that transcend traditional limitations.
Based on SUS and the participants’ feedback, we have observed that the designed SAR remote collaborative platform performs generally well for a training task. Nevertheless, further exploration is needed to address some of the disadvantages identified in our research. First, in the current study, our platform only supports sharing ATI from the remote site to the local SAR site. Thus, the local participants failed to express the information through visual communication cues and instead resorted to oral communication. We will explore how the ATI platform can provide local participants with the ability to create AAS or ATI by gesture interaction. Second, the system uses a fixed camera to perceive the state of the scene. In such cases, the remote expert may not always understand the local work environment. Therefore, we aim to utilize multiple cameras to enhance the remote expert's visibility of the task station. Third, In the research, we only explored commonly used four types of visual cues (e.g., arrows, circles, ellipses, straight lines) and how to affect SAR remote collaboration on training tasks. The engaging results showed that our direction was right. Next, we will improve the prototype platform and enhance its flexibility by supporting more types of visual cues. Finally, the platform provides remote participants with the information using video streams, which lack the depth information. And recently many studies have concentrated on supporting 3D information to enhance AR remote collaboration[11, 57, 58]. Consequently, we plan to reconstruct and share the 3D local scenario using the depth sensors. In practice, we’re trying to perform the related work as shown in Wang et al[15].
In this paper, we designed and developed a novel SAR remote collaborative platform for training tasks to explore the effects of converting AAS into standard symbols. To achieve this goal, we described in detail the platform architecture and key technologies (e.g., creating AAS, and converting AAS into standard symbols). Then we conducted a formal user study with two prototype platforms (e.g., the AAS interface, and the ATI interface). It should be noted that the ATI interface is based on the AAS interface. Furthermore, the experimental findings generally support four initial hypotheses to some extent concerning general collaborative experience (H2), usability based on ranking (H3), and preference (H4). Although the ATI condition wouldn’t provide faster performance than the AAS condition, it has a better user experience in terms of H2, H3, and H4, indicating that the ATI interface has a favorable impact on remote collaborative tasks. The engaging results verified our research direction was right and had a certain reference value for further research.
Our results synthesize previous research regarding how to convey information in AR for remote collaboration purposes. Specifically, our research is based on the fact that participants tend to sketch arrows, circles, and outlines for expressing the action-oriented instructions. However, these basic types may not be sufficient to accurately communicate the specific actions required. For a remote collaboration application in the industry, oral communication between team members could be utilized to serve as a critical supplement, but there is a risk of misinterpretations and language barriers. Our work can partially address this issue. Most related works[1, 11, 12, 30, 31] manifested that the SAR interface and the PC interaction interface are one of the most commonly used choices in the local site and the remote site, respectively. Consequently, we designed a SAR remote collaborative platform by making good use of their advantages. As far as we know, this is one of the first AR remote collaborative systems converting freeform sketches into standard symbols. Four hypotheses indicated that ATI-based cues have a positive rule in remote collaborative training tasks, especially, for both local and remote participants with respect to general collaborative experience, and usability based on ranking. Additionally, most local/remote participants prefer the ATI interface compared with the AAS interface corresponding to the results of users’ preferences. More importantly, this research pays the foundational way for creating AAS/ATI cues based on gesture-based interaction to improve remote collaboration.
Author Contributions: Xiuling Li finished the draft; Fusheng Li and Ruikun Zhang improved the paper. Moreover, Peng Wang designed the prototype system. Limin Yue and Huan Yan collected and analyzed experimental data.
Funding: This research was sponsored by the 2022 annual key research and development and promotion projects in Henan Province (222102220065), Special Project of the Center for Scientific Research and Development of Higher Education Institutions of the Ministry of Education (ZJXF2022006) and General Project of Chongqing Natural Science Foundation(CSTB2022NSCQ-MSX1153). We would also like to thank members of Zhengzhou Railway Vocational & Technical College for their participation in the experiment.
Data Availability Statement: No.
Conflicts of Interest: The authors declare no conflict of interest.
- Fidalgo, C. G., Yan, Y., Cho, H., Sousa, M., Lindlbauer, D., & Jorge, J. 2023. A Survey on Remote Assistance and Training in Mixed Reality Environments. IEEE T VIS COMPUT GR, 29(5), 2291-2303.
- Laverty, E.; Rahman M. A.; Jin Y.; Maropoulos P. G.; Mohamad E.; Salleh M. R.; Rahman A. A. A. Development of remote collaboration platform for manufacturing environment using mixed reality. In Proceedings of International Conference on Design and Concurrent Engineering 2023 & Manufacturing Systems Conference 2023. September 1-2, 2023, Hiyoshi Campus, Keio University, Japan & Online, 1-11.
- Marques B., Silva S., Maio R,. Dias P., Santos B. S. Guidelines for Designing Mixed Reality Solutions in Remote Scenarios. In Proceedings of HCI International 2023 Posters pp 247–252. Communications in Computer and Information Science, vol 1836. Springer.
- Marques, B., Silva, S., Maio, R., Vale Costa, L., Dias, P., & Sousa Santos, B. (2023, July). Remote Work Is Here to Stay! Reflecting on the Emerging Benefits of Mixed Reality Solutions in Industry. In International Conference on Human-Computer Interaction (pp. 253-260). Cham: Springer Nature Switzerland.
- Dinh, A., Yin, A. L., Estrin, D., Greenwald, P., & Fortenko, A. (2023). Augmented Reality in Real-time Telemedicine and Telementoring: Scoping Review. JMIR mHealth and uHealth, 11, e45464.
- Wang, Y., Wang, P., Luo, Z., & Yan, Y. (2022). A novel AR remote collaborative platform for sharing 2.5 D gestures and gaze. INT J ADV MANUF TECH, 119(9-10), 6413-6421.
- Marques, B., Ferreira, C., Silva, S., Dias, P., & Santos, B. S. (2023). Is social presence (alone) a general predictor for good remote collaboration? comparing video and augmented reality guidance in maintenance procedures. Virtual Reality, 1-14.
- Marques, B., Silva, S., Alves, J., Rocha, A., Dias, P., & Santos, B. S. (2022). Remote collaboration in maintenance contexts using augmented reality: Insights from a participatory process. International Journal on Interactive Design and Manufacturing (IJIDeM), 1-20.
- Feng, S., He, W., Zhang, X., Billinghurst, M., & Wang, S. (2023). A comprehensive survey on AR-enabled local collaboration. Virtual Reality, 1-26.
- Wang, Z., Bai, X., Zhang, S., Billinghurst, M., He, W., Wang, P., Lan W., Min H., Chen, Y. (2022). A comprehensive review of augmented reality-based instruction in manual assembly, training and repair. ROBOT CIM-INT MANUF, 78, 102407.
- Wang, P., Bai, X., Billinghurst, M., Zhang, S., Zhang, X., Wang, S., He W., Yan Y., Ji, H. (2021). AR/MR remote collaboration on physical tasks: a review. ROBOT CIM-INT MANUF, 72, 102071.
- Wang, P., Zhang, S., Billinghurst, M., Bai, X., He, W., Wang, S., Sun M., Zhang, X. (2020). A comprehensive survey of AR/MR-based co-design in manufacturing. ENG COMPUT-GERMANY, 36, 1715-1738.
- Chang, E., Lee, Y., & Yoo, B. (2023). A User Study on the Comparison of View Interfaces for VR-AR Communication in XR Remote Collaboration. INT J HUM-COMPUT INT, 1-16.
- Wang, P., Bai, X., Billinghurst, M., Zhang, S., Han, D., Sun, M., Wang Z., Lv H., Han, S. (2020). Haptic feedback helps me? A VR-SAR remote collaborative system with tangible interaction. INT J HUM-COMPUT INT , 36(13), 1242-1257.
- Wang, P., Wang, Y., Billinghurst, M., Yang, H., Xu, P., & Li, Y. (2023). BeHere: a VR/SAR remote collaboration system based on virtual replicas sharing gesture and avatar in a procedural task. Virtual Reality, 1-22.
- Wang, P., Bai, X., Billinghurst, M., Zhang, S., Wei, S., Xu, G., He W., Zhang X., Zhang, J. (2021). 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration. MULTIMED TOOLS APPL, 80, 31059-31084.
- Silvestri, L., Forcina, A., Introna, V., Santolamazza, A., & Cesarotti, V. (2020). Maintenance transformation through Industry 4.0 technologies: A systematic literature review. COMPUT IND, 123, 103335.
- Feng, S., He, X., He, W., Li, R., & Zhang, S. (2023). First-and Third-person Perspectives in the AR-assisted Blind Area Assembly.
- Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–149). American Psychological Association.
- Yan, Y., Bai, X., He, W., Wang, S., Zhang, X., Wang, P., Liu L., Zhang, B. (2023). Can you notice my attention? A novel information vision enhancement method in MR remote collaborative assembly. INT J ADV MANUF TECH, 1-23.
- Zhang, X., Bai, X., Zhang, S., He, W., Wang, S., Yan, Y., Yu Q., Liu, L. (2023). A novel MR remote collaboration system using 3D spatial area cue and visual notification. J MANUF SYST, 67, 389-409.
- Wang, J., Fan, X., Zhu, Y., Yang, X., & Yin, X. (2022). Cross-platform AR annotation for assembly-design communication in pipe outfitting. INT J ADV MANUF TECH, 121(7-8), 4981-4998.
- Wang, P., Zhang, S., Bai, X., Billinghurst, M., He, W., Sun, M., Chen Y., Lv H., Ji, H. (2019). 2.5 DHANDS: a gesture-based MR remote collaborative platform. INT J ADV MANUF TECH, 102, 1339-1353.
- Gurevich, P., Lanir, J., & Cohen, B. (2015). Design and implementation of teleadvisor: a projection-based augmented reality system for remote collaboration. COMPUT SUPP COOP W J, 24, 527-562.
- Lanir, J., Stone, R., Cohen, B., & Gurevich, P. (2013, April). Ownership and control of point of view in remote assistance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2243-2252).
- Scurati, G. W., Gattullo, M., Fiorentino, M., Ferrise, F., Bordegoni, M., & Uva, A. E. (2018). Converting maintenance actions into standard symbols for Augmented Reality applications in Industry 4.0. COMPUT IND, 98, 68-79.
- Zhang, X., Bai, X., Zhang, S., He, W., Wang, P., Wang, Z., Yan Y., Yu, Q. (2022). Real-time 3D video-based MR remote collaboration using gesture cues and virtual replicas. INT J ADV MANUF TECH, 121(11-12), 7697-7719.
- Marques, B., Silva, S., Teixeira, A., Dias, P., & Santos, B. S. (2022). A vision for contextualized evaluation of remote collaboration supported by AR. COMPUT GRAPH-UK, 102, 413-425.
- Fang, W., Chen, L., Zhang, T., Chen, C., Teng, Z., & Wang, L. (2023). Head-mounted display augmented reality in manufacturing: A systematic review. ROBOT CIM-INT MANUF, 83, 102567.
- Druta, R., Druta, C., Negirla, P., & Silea, I. (2021). A review on methods and systems for remote collaboration. Applied Sciences, 11(21), 10035.
- De Belen, R. A. J., Nguyen, H., Filonik, D., Del Favero, D., & Bednarz, T. (2019). A systematic review of the current state of collaborative mixed reality technologies: 2013–2018. AIMS Electronics and Electrical Engineering, 3(2), 181-223.
- Thomas, B. H., Marner, M., Smith, R. T., Elsayed, N. A. M., Von Itzstein, S., Klein, K., ... & Suthers, T. (2014, November). Spatial augmented reality—A tool for 3D data visualization. In 2014 IEEE VIS international workshop on 3DVis (3DVis) (pp. 45-50). IEEE.
- Wang, Z., Bai, X., Zhang, S., Billinghurst, M., He, W., Wang, Y., Han D., Chen G., Li, J. (2021). The role of user-centered AR instruction in improving novice spatial cognition in a high-precision procedural task. ADV ENG INFORM, 47, 101250.
- Higuch, K., Yonetani, R., & Sato, Y. (2016, May). Can eye help you? Effects of visualizing eye fixations on remote collaboration scenarios for physical tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5180-5190).
- Wang, P., Zhang, S., Bai, X., Billinghurst, M., Zhang, L., Wang, S., Han D., Lv H., Yan, Y. (2019). A gesture-and head-based multimodal interaction platform for MR remote collaboration. INT J ADV MANUF TECH, 105, 3031-3043.
- Wang, P., Bai, X., Billinghurst, M., Han, D., Zhang, S., He, W., Zhang X., Yan, Y. (2019). I’m tired of demos: an adaptive MR remote collaborative platform. In SIGGRAPH Asia 2019 XR (pp. 17-18).
- Wang, P., Bai, X., Billinghurst, M., Zhang, S., Han, D., Lv, H., He W., Yan Y., Zhang X., Min, H. (2019, October). An MR remote collaborative platform based on 3D CAD models for training in industry. In 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-adjunct) (pp. 91-92).
- Gattullo, M., Evangelista, A., Uva, A. E., Fiorentino, M., & Gabbard, J. L. (2020). What, how, and why are visual assets used in industrial augmented reality? A systematic review and classification in maintenance, assembly, and training (from 1997 to 2019). IEEE T VIS COMPUT GR, 28(2), 1443-1456.
- Kim, S., Huang, W., Oh, C. M., Lee, G., Billinghurst, M., & Lee, S. J. (2023). View Types and Visual Communication Cues for Remote Collaboration. Computers, Materials and Continua, 4363-4379.
- Bernardo Marques, Carlos F erreira, Samuel Silva, P aulo Dias, Beatriz Sousa Santos (2023): Insights on the Impact of Gender and T echnological Expertise in Augmented Reality Remote Collaboration. In: Proceedings of the 21st European Conference on Computer-Supported Cooperative W ork: The International V enue on Practice-centred Computing on the Design of Cooperation T echnologies - P osters, Reports of the European Society for Socially Embedded T echnologies.
- Marques, B., Teixeira, A., Silva, S., Alves, J., Dias, P., & Santos, B. S. (2022). A critical analysis on remote collaboration mediated by Augmented Reality: Making a case for improved characterization and evaluation of the collaborative process. COMPUT GRAPH-UK, 102, 619-633.
- Cidota, M., Lukosch, S., Datcu, D., & Lukosch, H. (2016). Comparing the effect of audio and visual notifications on workspace awareness using head-mounted displays for remote collaboration in augmented reality. Augmented Human Research, 1, 1-15.
- Kim, S., Lee, G., Billinghurst, M., & Huang, W. (2020). The combination of visual communication cues in mixed reality remote collaboration. J MULTIMODAL USER IN, 14, 321-335.
- Kim, S., Billinghurst, M., & Lee, G. (2018). The effect of collaboration styles and view independence on video-mediated remote collaboration. COMPUT SUPP COOP W J, 27, 569-607.
- Kim, S., Billinghurst, M., Lee, C., & Lee, G. (2018). Using freeze frame and visual notifications in an annotation drawing interface for remote collaboration. KSII Transactions on Internet & Information Systems, 12(12).
- Lorenz, M., Knopp, S., & Klimant, P. (2018, October). Industrial augmented reality: Requirements for an augmented reality maintenance worker support system. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (pp. 151-153). IEEE.
- Stork, S., & Schubö, A. (2010). Human cognition in manual assembly: Theories and applications. ADV ENG INFORM, 24(3), 320-328.
- Mohr, P., Kerbl, B., Donoser, M., Schmalstieg, D., & Kalkofen, D. (2015, April). Retargeting technical documentation to augmented reality. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 3337-3346).
- Engelke, T., Keil, J., Rojtberg, P., Wientapper, F., Schmitt, M., & Bockholt, U. (2015, March). Content first: a concept for industrial augmented reality maintenance applications using mobile devices. In Proceedings of the 6th ACM multimedia systems conference (pp. 105-111).
- Kritzler, M., Murr, M., & Michahelles, F. (2016, November). Remotebob: support of on-site workers via a telepresence remote expert system. In Proceedings of the 6th International Conference on the Internet of Things (pp. 7-14).
- Webel, S., Bockholt, U., Engelke, T., Gavish, N., & Tecchia, F. (2011). Design recommendations for augmented reality based training of maintenance skills. In Recent trends of mobile collaborative augmented reality systems (pp. 69-82). Springer New York.
- Wang, Z., Zhang, X., Li, L., Zhou, Y., Lu, Z., Dai, Y., Liu C., Su Z., Bai X., Billinghurst, M. (2023). Evaluating visual encoding quality of a mixed reality user interface for human–machine co-assembly in complex operational terrain. ADV ENG INFORM, 58, 102171.
- Marques, B., Silva, S., Rocha, A., Dias, P., & Santos, B. S. (2021, March). Remote asynchronous collaboration in maintenance scenarios using augmented reality and annotations. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 567-568). IEEE.
- Wang, P., Bai, X., Billinghurst, M., Zhang, S., He, W., Han, D., Wang Y., Min H., Lan W., Han, S. (2020). Using a head pointer or eye gaze: The effect of gaze on spatial AR remote collaboration for physical tasks. INTERACT COMPUT, 32(2), 153-169.
- Wang, P., Zhang, S., Bai, X., Billinghurst, M., He, W., Wang, S., Zhang X., Du J., Chen, Y. (2019, March). Head pointer or eye gaze: Which helps more in MR remote collaboration?. In 2019 IEEE conference on virtual reality and 3D user interfaces (VR) (pp. 1219-1220). IEEE.
- Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395.
- Bai, H., Sasikumar, P., Yang, J., & Billinghurst, M. (2020, April). A user study on mixed reality remote collaboration with eye gaze and hand gesture sharing. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1-13).
- Yang, J., Sasikumar, P., Bai, H., Barde, A., Sörös, G., & Billinghurst, M. (2020). The effects of spatial auditory and visual cues on mixed reality remote collaboration. J MULTIMODAL USER IN, 14, 337-352.
- https://unity.com
- http://www.wampserver.com.
- https://github.com/shimat/opencvsharp
- https://pc.weixin.qq.com
- https://unityassets4free.com/vectrosity/
- https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.htmlhttps://measuringu.com/sus