Augmented Reality Application for Laptop Assembly with Assembly Complexity Study

Augmented Reality (AR) in the assembly process will improve the user's experience by providing interactive instructions in real time. However, no previous application of AR guided assembly for laptops with a high level of assembly complexity has been developed. The research aims to develop an AR guided assembly application to provide instruction on the assembly of a laptop. The assembly complexity of the laptop was also investigated. The development of the AR application involves the creation of model target, 3D models and animations, and the development of user interface. The laptop assembly consists of ten steps. Each step comprises animated 3D models and text detailing the assembly instructions. Speech recognition has been used to navigate the assembly sequence. The AR application has successfully been developed for laptop assembly with an assembly complexity of 6.63. With the developed application, the performance of the laptop assembly can be accelerated.


Introduction
Augmented Reality (AR) is a human-computer interaction technology that superimposes the physical world with computer-generated visual information for a better user experience [1].The benefits of AR are being explored in the manufacturing industry as the software and hardware of AR are upgraded [2].AR technology has an impact on the manufacturing industry in product development and manufacturing processes by shortening the cycle time, improving product quality, and reducing costs [3].Although AR development is still in the early stage, the AR is effective in various areas, like facility layout planning [4], highlight weld placement [5], assembly, packaging [6], and maintenance [7].
Assembly includes the handling and insertion of parts that match together to form an added value product.Despite the automation of modern manufacturing, the manual assembly by human operators remains significant in performing assembly tasks.The human operators are highly flexible and adaptable and essential for customized production.In assembly planning, comprehensive instruction includes the assembly steps, the sequence of the steps, secure part, and methods is necessary [8].In tradition, instruction is usually recorded as a paper manual in texts or drawings.However, the right representation of the different parts and the assembly steps are not an easy task [8].Moreover, the methods are more time-consuming, and prone to errors, as assembly tasks get more complex.
For ease of manual assembly, research on computer-based tools, such as the Augmented Reality (AR) is stimulated.The AR technology providing assembly information step-by-step in real-time instructions.
Compared to the paper manual, AR increases operators' productivity and users well accept to it [7].The use of AR reduces the cognitive load and switching focus in attending the manual and the working space.
Switching focus is time-consuming and leads to an increase in assembly time and thus decreases efficiency.
AR guided assembly enables operators to focus on assembly activities at hand without paying attention to finding the necessary assembly instructions [9].Henderson and Feiner [10] established the AR guided assembly application that outperformed traditional paper-based manuals with an error reduction of over 60 percent and a 100 percent increase in assembly speed.Besides, the AR implementation raises the confidence of the operator to repeat the assembly task and to adapt the developed skills to other tasks [11].
In the literature, several studies have demonstrated the application of AR in the assembly domain.
Wang et al. [12] summarized the AR assembly studies and forecasting future research and development trends.Suárez-Warden et al. [13] conducted a study to practice aircraft assembly using AR.The findings demonstrate that the AR-guided assembly of complicated parts requires less time than the standard assembly.The first assembly method integrated with AR has been established by Thomas and David [14] to assist personnel in the assembly of wire bundles in aircraft.To show the necessary way to link the cable to the wire bundle, a simple wire graph has been created.The AR demonstrator was developed by Reiners et al. [15] to guide the worker for assembly of the door lock in the car door.The AR application with head-mounted display (HMD) for the assembly of computer motherboard parts was introduced Baird and Barfield [16].The participants of the survey were asked to execute the assembly activities using a paper manual, an electronic manual, and the developed AR based HMD system.The results indicate that the assembly task done with HMD-based AR is quicker with fewer errors.The AR application for assembly guidance was introduced by Pang et al. [17] where virtual numbers of the assembly sequence were overlaid on the assembly parts.Miller et al. [18] have developed a Microsoft HoloLens 1 AR application to assemble the mock aircraft wing.
Assembly complexity is an effective tool for designers to measure the assembly process's complexity.Therefore, it is useful to develop AR's application in assembly with a complexity assembly analysis.Such research, however, had not been published previously.This paper aims to develop an AR-guided application to provide instruction for a laptop assembly and studied the assembly complexity of the laptop.The assembly is described by a sequence of assembly procedures to fix together the laptop parts.The paper is organized into six sections.Section 2 describes the state-of-art of the AR guided assembly study is reviewed and the technology gap is addressed.The calculation and the results of the product assembly complexity of the laptop are given in Section 3 and Section 4 respectively.Section 5 explains the AR application development.Section 6 details the visual features created in the AR application for each assembly step and Section 7 details the conclusion.

State-of-art of the AR guided assembly
There is diversity among the development of AR application in assembly with respect to the technologies such as tracker, visual features, and input modalities.The tracker enables to register and trigger the augmented objects to be superimposed in the correct position in the real environment.The tracker is divided into vision based and sensor based [19].The vision-based trackers include feature-based, modelbased, and marker-based trackers.The trends of the implementation of tracking method have been discussed Refs. [12].The marker-based tracker, such as rectangular pre-defined fiducial markers has been widely implemented by many researchers in the development of the application of AR in assembly [12].Radkowski et.al.[11] explored various kinds of visual features for various assembly operations in the manual assembly of the axial piston engine with AR.The fiduciary markers were attached to the parts and the work table.A HMD-based demonstrator was developed by Reiners et al. [15] to navigate operators to assemble the door lock in the car door.The fiducial markers were added to the door lock.Westerfield et al. [20] have established an AR system for manual assembly training of motherboard computers.Fiducial markers have been placed around the motherboard.They observed that the fiducial marker had low resistance to occlusion, low accuracy, and obtrusive markers.A more advanced marker such as a physical object in a real world has been used to develop an AR application to assist maintenance operators [21].Rather than having the fiducial markers attached to the base part, the marker is the base part itself.Object tracking is much preferred to image tracking as object tracking techniques are able to track 3D objects having very detailed and extensive features.
In the AR guided assembly, the simulated visual features are overlaid on the physical assembled parts.
Visual features refer to animated 3D models, text, and annotations.Visual features are essential to improving user interpretation of step-by-step assembly operations.Pang et al. [17] designed an AR guided assembly with overlaid numbers on the actual assembled part to display the sequence of assembly.Reinhart and Patron [22] established an AR guided assembly by displaying assembly details via 3D models and texts.Webel et al. [23] created an AR-based platform to train assembly and maintenance skills.The AR application incorporates a simplified 3D model to indicate the assembled part and the visual objects that display the handling details.However, no 3D animation models were created in the studies described above.Animated 3D models make the assembly process easier to understand relative to standard user manuals [24].
Radkowski et al. [11] studied two types of visual features used in the AR guided manual assembly of the axial piston motor.They compared visual features in text and animation, text and drawing.The findings showed that visual features in text and animation outperform those in text and drawing.The visual features are superimposed over the part to indicate the secure location of the assembled part.In addition, more comprehensive assembly information, such as the part-secure method should be given to improve users' comprehension of assembly operations.However, the application of AR guided assembly with 3D model animations and the information of part secure method has been neglected in the past.
As the user interface of the AR guided assembly, different input modalities are applied.The input modalities include the keyboard, touchscreen, pointing device, gesture control, and speech recognition.Input modalities are necessary to navigate the linear sequence of visual features.Previous or next data is the input of the modalities.This allows human-computer interaction where the recall on-demand the assembly sequence whenever required.Hořejší [25] introduced an AR system for the virtual training of parts assembly using the keypad as an input modality.Research on various features for AR-based assembly has been released by Radkowski et al. [11] using the keyboard as an input modality.Reiners et al. [15] established an AR-based demonstrator to assist the operator through the assembly of the door lock in the car door with speech recognition as the input modality.In assembly tasks, speech recognition is preferred for operators to concentrate on assembly activities and eliminate the attention on searching keys on the keyboard.Wasted movement, such as pressing keys on the keyboard or keypad is omitted and the assembly time is therefore reduced.
Past research has extensively expanded the application of the AR guided assembly.However, an ARguided assembly development has not been carried out in the past with a product assembly complexity analysis.The study is intended to develop an AR guided assembly application to provide laptop assembly planning with a complexity analysis of the product assembly.The AR guided assembly was built using the base part as the marker.Visual features such as animated 3D models, texts, and annotations were created to describe the handling, and insertion as well as the secure method.

Product Assembly Complexity Study of a laptop
The product assembly complexity   is calculated using Eqn (1) where   refers to the product assembly complexity index,   and   refers to the number of unique parts and fasteners respectively.For Asus G750JW laptop, the   and   are 10 and 4 respectively.Fig. 1 shows the laptop which consists of 10 parts.The   and   refers to the total amount of parts and fasteners respectively.The   and   of the laptop are 10 and 24 respectively.Eqn.(1) to Eqn. ( 5) are published in Refs. [26].
The product complexity index   for the laptop is calculated using Eqn ( 2) where   represents the percentage of the  th unique parts.The   for each unique part are 0.1 since there are 10 unique parts in the laptop.For each assembly part, the weighted average value of the assembly complexity factors   is given by Equation ( 3) where  ℎ refer to average handling complexity factor for each part and   refer to average insertion complexity factor for each part.The  ℎ and   are calculated using Eqns.( 4) and (5).The  ℎ represents the average handling difficulty factor for each part from Table 1 and   represents the average insertion difficulty factor for each part from Table 2 for insertion.
where  refers to the number of handling attributes of each assembly part.
where  refers to the number of insertion attributes of each assembly part.
From equation (1) to equation ( 5), the assembly complexity of a product is influenced by handling and insertion attributes which are relating to the assembly parts.The number of parts and the number of fasteners necessary to secure the parts would also affect it.

Result and discussion for product assembly complexity of a laptop
There are 10 parts of the laptop.The average handling complexity factor  ℎ for each part is calculated and is tabulated in Table 3.The average handing difficulty factor  ℎ for each handling attribute in Table 3 are obtained from Table 1 and the  ℎ is calculated using Eqn.(4).The average insertion complexity factor calculation for each part is also presented in Table 4.The average insertion difficulty factor   of each handling attribute are extracted from Table 2 and the   is calculated using Eqn.(5).
By substitute the  ℎ value and  ℎ value from Table 3 and   value and   value from Table 4 for each part into Eqn (3), the weighted average values of the assembly complexity factor   for each part is calculated.The results of   is given in Table 4.By substituting the results of   in Table 4 into Eqn.
(2) the assembly complexity index   of the laptop is calculated as 0.694.The product assembly complexity   of the laptop is then determined by substitute   value of 0.694 into Eqn.(1).
The   of the laptop is 6.63.Step 2. The voice recognition was then used to navigate the linear sequence of assembly using keywords like "Next" or "Back".In Step 5, the AR App is exported to the computer's Windows platform.

AR application development for assembly of laptop
The AR application development for the laptop assembly is presented.A total of 10 laptop assembly procedures are given in Fig. 5.To explain the assembly instructions, each operation will be directed by animations of 3D models and texts.Certain procedures are accompanied by an annotation arrow.Speech recognition was used between assembly procedures to continue or to wind up.The laptop scene view with 3D models for Procedure 1 to 4 and Procedures 6 to 10 is shown in Fig. 6 to Fig. 14.Every component is indicated with (r) or (v) in Fig. 6 to Fig. 14.The (r) shows the real components and (v) indicated the virtually generated solid model using Computer-Aided Design (CAD) software.
Procedure 1 is the assembly of the motherboard with laptop sub-assembly.Assembly includes the motherboard insertion and the insertion and tightening of the five screws and connecting of the speaker cable to the connector on the motherboard.Fig. 6(a) displays the scene view with a slanting simulated motherboard.
To complete the insertion of the motherboard, the motherboard connectors are inserted in the connecter slots in the laptop base subassembly as seen in Fig. 6(b).Afterward, the motherboard was positioned at the assembled place as seen in Fig. 6(c).Following insert of the motherboard, Fig. 6(d) displays the scene view containing five virtual screws above its assembly position and a virtual speaker cable located in front of its connector on the motherboard.The screws were animated to be moving downwards vertically and are inserted and tightened in their respective holes.The speaker cable has been moved up until the connector of the speaker cable was inserted into the motherboard connector.The real speaker cable is indicated by an arrow to help the operator search the cable quickly.The scene view of complete motherboard assembly after Procedure 1 is seen in Fig. 6(e).
Procedure 2 is the assembly of the left cooling fan, whereby the cooling fan is inserted, two screws are inserted and tightened and the cable of the cooling fan is connected to the connector at the motherboard.
The scene view for a virtual cooling fan above its assembly position is shown in Fig. 7(a).The dashed line box in Fig. 7(a) to Fig. 13(a) shows the area to be enlarged and illustrated in the following subfigures for each figure.The virtual cooling fan was moved downward and positioned in its slot.The scene view of two virtual screws placed over its mounting position, with a virtual cooling fan cable located in front of the cable connector on the motherboard, is seen in Fig. 7(b).The screws were vertically moved downward and inserted and tightened in the holes on the cooling fan.Furthermore, before a cable connector of the cooling fan was inserted into the motherboard cable connector, the cooling fan cable was shifted horizontally to the side.A visual arrow shows the location of the real connector cable for ease of searching.Fig. 7(c) displays the scene view of the entire left cooling fan assembly after Procedure 2.
Procedure 3 includes inserting the Universal Serial Bus (USB) board with three screws inserted and tightened.The scene view of a virtual USB board placed above its assembly location is shown in Fig. 8(a).
The virtual USB board has been moved downward vertically and is positioned in the slot.The three virtual screws above their assembly point are displayed in Fig. 8 Procedure 5 is the right cooling fan assembly.The procedure is identical to Procedure 2, the left cooling fan assembly.Procedure 6 includes the insertion and tightening of the power board by two screws.
The view of a virtual power board above its assembly location is shown in Fig. 10 (a).To complete the insertion, the power board has been shifted downward vertically.The pins at the bottom of the power board were then inserted into the pin connector on the motherboard.The textual instruction led the task as the user has restricted vision of the pins.The scene view of two virtual screws placed above its assembly location is shown in Fig. 10(b).The screws were moved downwards vertically and inserted for tightening into the power board.The scene view of the complete power board assembly following Procedure 6 is shown in Fig. 10(c).Display (LCD) cables to the corresponding connectors of the motherboard.The scene view of a virtual panel kit above its assembling position is shown in Fig. 13(a).The panel kit was moved downwards vertically to its place of assembly.The screws as shown in Fig. 13(b) were then shifted downwards vertically to the holes on the kit brackets.After screw tightening, the panel kit was lifted for the next assembly step.WIFI cables and LCD cables were connected to the corresponding connectors on the motherboard via snapfit joint as shown in Fig. 13(c).The scene view of the entire panel kit assembly after Procedure 9 is shown in Fig. 13(d).
Procedure 10 is the assembly of a keyboard that includes the insertion of a keyboard, connection of the keyboard cables to a motherboard connector, and keyboard securing.Fig. 14 (a) displays a scene view of the virtual keyboard in a slanting position.The keyboard cable was connected to the connectors on the motherboard to complete the insertion.Fig. 14(b) shows the scene view of the keyboard at which it was positioned at the installed location.The keyboard edges are pressed such that they are snapfitted to the laptop base.Fig. 14(c) shows the scene view of the complete keyboard assembly described in Procedure 10.

Conclusion
An Augmented Reality (AR) application for the laptop assembly with product assembly complexity,   of 6.63 has been successfully developed.In the development of AR, object tracking technology has been introduced which can track 3D objects with detailed features.The assembly part in the physical world, the laptop base subassembly was used as a marker in triggering virtual 3D objects.The AR app is developed with animation of virtual 3D models to enable users to gain a deeper understanding of the assembly operation at each assembly step.To highlight the secure method of each assembly part, a visual feature such as virtual text was implemented.Input modality, such as speech recognition, has been built into the user interface for processing or rewinding between each assembly step.The AR application is demonstrated clearly through a display monitor. Exploded The laptop base subassembly in model target generator The assembly sequences of the laptop part

Fig. 2
Fig. 2 demonstrates the development of the AR application framework in five stages.The five stages include 3D assembly part modeling, model target creation to track objects, 3D model animation creation, development of the user interface to link assembly steps in sequence, and the export of applications to a computer.
(b).The screws were moved downwards and inserted for tightening into the holes on the USB board.The scene view of the complete USB Board assembly after Procedure 3 is shown in Fig.8(c).The Universal Serial Bus/Motherboard (USB/MB) cable connectors are inserted on the USB board as well as the motherboard in Procedure 4. The scene view of the virtual cable of USB/MB located in front of the assembly position is shown in Fig.9(a).The USB/MB cable was shifted until the cable connections are inserted into the connectors of the USB board and motherboard.The complete assembly of the USB/MB cable after Procedure 4 is shown in Fig. 9(b).

Procedure 7
is an audio board assembly involving audio board insertion, insertion and tightening of two screws, and the pressing of the audio board.The view of a virtual audio board in an inclined position is shown in Fig.11(a).The audio-board ports as seen in Fig.11(b) were inserted first to their respective laptop base slots, as shown in Fig.11(c) for completion of insertion.Fig.11(e)shows the scene view of a virtual screw positioned above its assembly location and an arrow.The arrow pointing towards the surface of the audio board indicates the location to perform pressing.The pressing allows the connector at the bottom of the audio board to be inserted into the motherboard connector.The screw was moved downward vertically and inserted and tightening in the hole on the audio board.The scene view of the complete audio board assembly described in Procedure 7 is shown in Fig.11(f).The scene view of the virtual screw which is located above its assembly position and an arrow is given in Fig.11(e).The arrow pointing on the surface of the audio board shows the position for pressing.The pressing enables the connector at the bottom of the audio board to be inserted into the motherboard connector.The screw has been moved downward vertically and is tightened in the hole of the audio board.Procedure 8 comprises the insertion of a Compact Disc Read-Only Memory (CD-ROM).The scene view of a virtual CD-ROM located in front of its assembly place is shown in Fig.12 (a).The CD-ROM was shifted horizontally to the right until it was connected to the motherboard.The complete assembly of a CD-ROM following Procedure 8 is shown in Fig.12 (b).Procedure 9 is the panel kit assembly, involving the insertion of a panel kit, insertion and tightening of screws, and the connections of the WIFI and Liquid Crystal

view of 10 parts in the laptop Figure 2 Flow chart of AR application framework development Figure 3 (
Figure 2

Table 3 :
Calculation of average handling complexity factor   for the assembly parts of laptop

Table 4 .
Calculation of average insertion complexity factor Ci for the assembly parts of laptop