A closed-loop brain-computer interface with augmented reality feedback for industrial human-robot collaboration

Industrial human-robot collaboration (HRC) aims to combine human intelligence and robotic capability to achieve higher productiveness. In industrial HRC, the communication between humans and robots is essential to enhance the understanding of the intent of each other to make a more fluent collaboration. Brain-computer interface (BCI) is a technology that could record the user’s brain activity that can be translated into interaction messages (e.g., control commands) to the outside world, which can build a direct and efficient communication channel between human and robot. However, due to lacking information feedback mechanisms, it is challenging for BCI to control robots with a high degree of freedom with a limited number of classifiable mental states. To address this problem, this paper proposes a closed-loop BCI with contextual visual feedback by an augmented reality (AR) headset. In such BCI, the electroencephalogram (EEG) patterns from the multiple voluntary eye blinks are considered the input and its online detection algorithm is proposed whose average accuracy can reach 94.31%. Moreover, an AR-enable information feedback interface is designed to achieve an interactive robotic path planning. A case study of an industrial HRC assembly task is also developed to show that the proposed closed-up BCI could shorten the time of user input in human-robot interaction.


Introduction
In recent years, industrial human-robot collaboration (HRC), which is a promising production paradigm that enables human and robot work side-by-side to jointly complete the same production goals, has enjoyed the more and more interest from industry and academia. Remanufacturing, defined as the process of returning a used product to at least original performance and at least the equal performance of the newly manufactured produce [1], is one of the research areas of industrial HRC. The potential of industrial HRC motivates developing innovative solutions to improve the autonomous degree of remanufacturing, especially the disassembly process which is the bottleneck of autonomous remanufacturing due to its traditional manual mode. Disassembly contains a series of complex operations, including unscrewing, removal of pins from holes, and separation of press-fit components. To deploy an industrial HRC setting into a disassembly process, many pieces of research mainly focus on disassembly planning [2][3][4], which aim at improving the coordination in the collaborative disassembly tasks to promote productivity. Besides that, an efficient and intuitive communication channel, which is essential for humans and robots to understand each other's intent, should be built to ensure the frequency and productivity of the collaborative disassembly tasks. In the earlier literature, various modal human-robot interfaces have been introduced into this context, such as gesture recognition [5], haptic interaction [6], and speech processing [7], to fulfill this end. The above interaction methods are more or less based on the human limb movements or verbal actions which might be difficult to ensure its robustness in a relatively uncertain environment (e.g., the visually obstructed or noisy environment).
With the last advances in neuroscience and brain-imaging technologies, some researchers have proposed the potential application of brain-computer interface (BCI) as an auxiliary channel for existing interaction solutions to support the industrial HRC [8,9]. Brain-computer interface (BCI) is a technology that monitors the activity of the human's central nervous system (CNS) to interpret the users' intention into readable information [10]. With BCI, users can explicitly manipulate their brain activity instead of limb movements to produce signals that control external machines, which is a more direct and natural way to convey their intention [11]. However, it is still challenging for BCI to control a robot with a high degree of freedom because of the limited number of mental states that could be classified in the existing BCIs which is unable to generate the high-dimensional commands. To overcome this problem, a potential solution can be to offer sufficient information feedback for the BCI users who can then perform their limited types of input to accomplish the generation of complex commands.
In this paper, we introduce augmented reality feedback to achieve a closed-loop BCI to control the robot in an industrial HRC scenario. Firstly, from a practical point of view, we use the voluntary eye blinks, which can be easily performed without intensive user training and can generate relatively strong signals with stable patterns, as the input of the BCI user to communicate with the robot. Secondly, an optical see-through (OST) AR headset is used to provide contextual visual feedback according to the input of BCI user, which can achieve the complex robotic operation, such as to specify a trajectory of robot movement. To verify the effectiveness of the proposed method, we also deploy a BCI-AR interface in an industrial HRC task and compare its performance with [12], a hand gesture-based AR interface.
The main contribution of this paper can be listed as follows. Firstly, we propose an online multiple voluntary eye blink detection algorithm based on the Blink algorithm [13]. Secondly, we combine the BCI and AR technology for more efficient human-robot communication.
The rest of the paper is organized as follows. Section 2 reviews previous work in the area of BCI and its combination with AR for human-robot communication. Section 3 gives an overview of the closed-loop BCI system. Section 4 describes the proposed EEG-based voluntary blinks online detection approach, while the design of the AR-enabled feedback interface is presented in Section 4. In Section 5, the performance analysis of the proposed detection algorithm and a case study in an industrial HRC task is developed. Conclusions and future work are summarized in Section 7.

Related work
Historically, the applications of BCI mainly focus on the assistance for people with motion disabilities, e.g., wheelchair [14], home automation [15], and word typing [16]. Focusing on one form of noninvasive BCI, electroencephalography (EEG), the brain activity is monitored as electrical signals measured on specific areas of the BCI user's scalp through the electrodes. In the past decades, many paradigms of EEGbased BCI have been developed, such as steady-state visual evoked potentials (SSVEPs) [17], P300 [18], and motorimagery [19]. These paradigms either require extra visual stimulus (SSVEP and P300) or require intensive user training (motor-imagery) which impose a challenge in the practical application for non-disabled users in some practical scenarios, like the industrial HRC.
In EEG-based BCI, eye blinks are generally seen as artifacts that should be removed before the next EEG signal processing. The eye blinks are typically classified into three categories: one is a spontaneous (or normal) eye blink which occurs frequently, another is a reflexive eye blink which is evoked by an external stimulus (e.g., some hazard situations), and the other is a voluntary eye blink which is caused by intentional eye closing [20]. The voluntary eye blink, which can be seen as the intentional action of the BCI users and does not require external stimulus nor intensive user training, has been used as the trigger signal of BCI in some research [21][22][23]. In the above view of the advantages of the voluntary eye blink's application in BCI, in this paper, it is used as the user input of the proposed closed-loop BCI interface.
In the EEG-based eye blink detection, the algorithms are mainly categorized into (1) threshold-based approaches, (2) template matching-based approaches, and (3) learning-based approaches. As for the threshold-based approaches, [24] proposed a multi-window summation of derivatives approach whose performance was compared with the root mean square error (RMSE), dynamic time warping (DTW), and other approaches. [25] adopted a threshold-based peak detection to switch on/off the home lighting system. [26] proposed a power spectrum density (PSD) threshold-based method, where a moving window of PSD was compared to a threshold to detect eye blink pattern. These threshold-based approaches are of high sensitivity to the feature selection and the current threshold, which could diff highly across users and hardware. As for the template matching-based approaches, [27] proposed a dynamic positional warping (DPW)-based method, which is a variant of DTW, and its accuracy was between the DTWbased and the RMSE-based algorithms. The template matching-based approaches depend on the manual template inspection or the specific algorithm for the template generation. As for the supervised learning-based approaches, [28] proposed a support vector machine (SVM)-based eye blink identification algorithm with a moving window. [29] proposed an RBF network-based eye blink feature extraction method with a segmentation of a fixed window of 1 s. Such learning-based methods require user training and machine learning which also needs a lot of labeled training data. [13] proposed a fully automated unsupervised blink detection algorithm Blink based on waveform feature detection and cluster analysis, which do not need any user training or manual labeling requirements. However, this algorithm is focusing on the normal blink detection instead of the voluntary one. In this paper, inspired by Blink, we propose a multiple voluntary detection algorithm in a fully automated unsupervised manner.
In the combination of BCI and AR, most of the existing studies are based on the SSVEP or P300 paradigm, where, on the one hand, BCI offers a hand-free interaction paradigm for AR; on the other hand, the AR offers a channel to integrate and feedback the real-world information for BCI users [30]. In [31], focusing on Industry 4.0, the authors proposed a wearable monitoring system for inspection based on AR glasses and a single-channel BCI. The system applied the SSEVP paradigm, whose visual stimuli was displayed through the AR glasses while displaying the inspection information. Some other robotic applications also have been developed for assisting disabled people. [32] proposed a BCI-based telepresence mobile robot with a monitor-based AR displayer, where the monitor could tele-present the task-related information and the BCI could detect the P300 signal to determine the task selected by users. In [33], a BCI for high-level remote robotic control is proposed, which combined the motorimagery paradigm and the monitor-based AR. The monitor displays the objects that the robot could grasp for the user, and the user can perform different movements imaginary to select the object to be grasped. These researches are mainly focusing on the high-level control which is unable to specify the detailed configuration of the robot operation, like the path planning. In [34], an AR-based robot path planning interface has been proposed, which is based on a monitor-based AR device and a hand-held tool for inputting the planning command. A similar monitor-based interface proposed in [35] uses hand gestures as input with the recognition by a camera. In [36], an AR HMD (HoloLens) is used to develop an interactive robotic pathing programming with the 3D visual trajectory feedback and hand gesture inputs. In this paper, inspired by the AR-based robot path planning in [34][35][36], we propose an AR-enable robot path planning with the input from BCI and without any extra hand-held interaction tool and hand gesture inputs.

System architecture
The developed system makes full of the advantage of braincomputer interface (BCI) and augmented reality head-mounted display (AR HMD). On the one hand, the BCI brings a direct input channel for human to non-verbally convey their control intention to robot without any limb movement. On the other hand, the equipped AR HMD serves as a visual aid tool to provide human an on-site information feedback (the EEG detection result, the robotic motion trajectory, the robotic motion preview, etc.) via the visual elements which is an efficient way to improve the human's understanding to the motion intent of collaborative robot [37].
As shown in Fig. 1, the developed system consists of an EEG-based multiple voluntary blinks detector, an augmented reality feedback interface, and a bi-direction information pipeline.
In the EEG-based multiple voluntary blinks detector, the input EEG raw data is collected by OpenBCI Cyton [38] with three wet gold cup electrodes (one is on the Fp1 of international 10-20 electrode system (see Fig. 2) as signal source, and two are on the earlobes as reference electrodes) and is analyzed by the blink detection algorithm. The detector algorithm consists of pre-processing, feature extraction, and unsupervised clustering, which can online detect the user's voluntary eye blinks of two types (long blink and double blink) without any user training session. The voluntary blinks are regarded as the input of the user with control intention.
The augmented reality feedback interface consists of a control logic module, a robotic trajectory specification module, a motion preview module, and a command handling module. In the control logic module, the detected voluntary blinks' result is as the input signal of a finite state machine that can control the transition of the interaction status (including initial status, interactive path planning, motion preview, and command handling). The trajectory specification module and the motion preview module are both powered by the augmented reality rendering and are integrated into an interactive robotic path planning; the former uses a head pose capture built in the AR headset and the editable visual trajectory of the robot's end-effector as the spatial information of path planning from the user, and the voluntary eye blink can switch the interactive status to modify the path planning configuration; the latter is an AR simulation of the robot modified configuration which can give the human a preview of the robot's ongoing motion. In the command handler module, the modified robot path planning is translated into the robotic executable command, which enables human to on-the-fly specify the robot's motion target without programming.
The bi-direction information pipeline includes the signal captured from human (EEG data and head pose data) and translated into the commands for robot, and the AR elements (visual trajectory, 3D simulation, etc.) corresponding to the robot's status as the visual feedback for the human.

EEG-based voluntary eye blink online detection
Eye blinks can be classified as spontaneous (or normal), reflexive, and voluntary, among which, generally, the normal blinks and the voluntary blinks are the relatively common observable behavioral phenomenon [20]. As the voluntary blinks are driven by the intentional eye closing which can reflect human's intention to some extent, we focus on using it as the human's input of the human-robot communication channel. In this paper, we choose two kinds of specific voluntary blinks: (1) double blink and (2) long blink. Double blink includes two eye blinks with a short interval; long blink is the eye blink with a long duration of eye closing. The rest of this section will introduce how to implement the voluntary blink detection from the real-time EEG signal.

Eye blink patterns of EEG signal
When an eye blink occurs, the frontal EEG signal shows a typical trough pattern in the voltage-time domain. Figure 3 shows such EEG pattern of normal blink, double blink, and long blink at frontal electrodeposition (Fp1 according to the 10-20 electrode system) with two earlobe referenced electrodes. In the same-user case, these three kinds of eye blinks have similar amplitude with two stable points (which represents the starting moment of closing eyes and the ending moment of re-opening eyes) and one or two middle local minimum points (which represent the moment of full eye closure). Through the two stable points adjacent to the same local minimum point(s), the blink duration of different types can be computed; through two adjacent local minimum points, the blink interval can be obtained. According to the characteristics of EEG signal of eye blinks, we find that the amplitude, duration, and interval can be used to identify the different kinds of eye blinks: (1) the amplitude can be used to detect the local minimum points and corresponding stable points; (2) a short interval between local minimum points represents a double blink; (3) in the non-double blink case, a long duration represents a long blink. To achieve the above detection, the threshold of amplitude, interval, and duration should be determined to classify the different types of blinks.
The earlier literature [13] has found that the variability of amplitude occurs in the intra-subject cases (blinks of the same user in different periods) and inter-subject cases (blinks from different users). Thus, it is challenging to pre-determine a fixed threshold of amplitude. As for the duration and interval of blinks, the earlier literature [20] has found that the spontaneous blinks have a characteristic duration and interval distribution. To determine reasonable thresholds of blink interval and duration for voluntary blink detection, it is necessary to find out the boundary of interval and duration of spontaneous blinks. To this end, we collected an EEG dataset of 12 subjects (age range between 21 and 25; 1/3 are female and 2/3 are male) who performed video-watching tasks. As shown in Fig. 4, the subject, attaching wet electrodes with Ten20 conductive paste on the Fp1 and two earlobes, sat in front of two monitors, one plays videos and one visualizes the EEG data through the OpenBCI GUI [40]. The PC of the monitors is in the same LAN as the OpenBCI Cyton which collects and sends EEG data to the OpenBCI GUI through a WiFi shield.
The dataset then was processed by a normal blink detection algorithm Blink [13], and the detection result of subject 1 is shown in Fig. 5 (x-axis: time-domain in seconds; y-axis: The international 10-20 EEG electrode placement system [39] voltage-domain in microvolt; the gray shades indicate the all detected eye blinks and their widths represent the duration of each blink). The blink duration and interval of all blinks then was obtained. After that, we randomly sampled 70 sets of data from each subject for statistics, whose result is shown in Fig. 6. From the result, we find that the duration of the normal blinks from all subjects holds a median between 200 and 300ms with an upper limit lower than 500ms. As for blink interval, all medians are larger than 1s and the lower limit is around 700ms. With these findings, to clearly distinguish between normal and voluntary blink, we safely set the duration threshold to 600ms (a long blink should last over 600ms) and set the interval threshold to 600ms (two eye-closing actions of a double blink should perform within 600ms).

Multiple voluntary eye blink detection algorithm
The aim of the multiple voluntary eye blink detection algorithm is to recognize the long blink and the double blink from the EEG raw data without being disturbed by noise and the normal blink. Firstly, the patterns of the eye blinks of different kinds from a single subject in a short period (e.g., during data acquisition) have repeatability in waveform (e.g., the trough pattern) and voltage domain (e.g., the amplitude), which is the distinct feature compared to noise generally. Secondly, as mentioned in Section 4.1, the time-domain features of long blink, double blink, and normal blink are distinguishable, especially the blink duration and interval.
Given the above knowledge, based on a clustering-based unsupervised algorithm Blink [13] dedicated to detecting repeating normal blink patterns, we propose an extended algorithm for the multiple voluntary eye blinks (double blinks and long blinks). The algorithm is based on the assumption that (1) the EEG data to process contains more than one pattern of each type of eye blink, which requires the subjects to perform repeatedly all types of eye blink in the EEG recording period; and (2) there is no other repetitive noise with a similar waveform as eye blinks. The second assumption is easy to achieve as the EEG signal from the electrode at the forehead scalp is mainly affected by the motion of eyes, head, and muscles [41], all of which, except eye blink, are either without troughshaped EEG pattern or generally non-repetitive.
The proposed algorithm has some characteristics. Firstly, just only one EEG channel's data is required which helps to achieve the low-cost and ergonomics human-robot communication channel; secondly, just four parameters are needed (two Fig. 3 The typical EEG waveforms of normal blink, double blink, and long blink Fig. 4 The setup of EEG data collection thresholds related to the time-domain feature, and two thresholds related to the waveform feature), and these parameters are easy to be fine-tuned; thirdly, the targeted voluntary eye blinks are easy to be performed by users and thus the users are away from any training requirements.
Algorithm explanation: (1) after collecting the raw EEG data E, the first procedure is pre-processing which is to filter the high-frequent component of the raw data; (2) the first step of the algorithm is to detect the peaks of the filtered data (see Fig. 3), which are the local minimums with a threshold delta (=80μA, initially); (3) according to the intervals of the detected peaks, classify the detected peaks into double peak and single peak with a threshold interval=600ms which is determined in Section 4.1; (4) input the classified double peaks and single peaks into subroutine detect_blink respectively to obtain the final result of the double-blink detection T double and single-blink detection T single which include long blinks and normal blinks; (5) classify the long blinks T long from T single with the threshold duration=600ms which is determined in Section 4.1. So far, the T double and T long are the final output of the algorithm.
Subroutine detect_blink, following [13], detects the blinks based on time-domain feature and cluster analysis. Firstly, the Fourthly, combining the time-domain and voltage-domain features, the indexes index _ corr are clustered into two groups and the larger average power group is selected as the index of detected blinks (line 10). Finally, to detect the missing eye blink patterns, a new delta for detecting local minimums of the EEG signal is computed and the second pass of eye blink pattern detection is done (lines 11 to 13). Then, the start and end times of all detected eye blinks will be returned.

Online detection approach
The above algorithm is based on time-domain and voltagedomain feature extraction and cluster analysis, by which eye blink patterns can be detected in an unsupervised and trainingfree way. However, due to the characteristic of cluster analysis, it needs to collect EEG data over a relatively long period, in which multiple eye blinks of various types should occur. Thus, it is hard to perform in real time. To address that problem, we propose an online detection approach to make full use of the above algorithm, as shown in Fig. 7. The approach includes a calibration session lasting a period of t 0 which is done before the online detection. In the calibration session, an EEG dataset with repeated patterns of three types of eye blinks (long blink, double blink, and normal blink) is collected. The dataset then is input to the eye blink detector based on the proposed algorithm, after which the detection n times eye blinks and the features (including signal segments Sig 0 = [sig 1 , sig 2 , …, sig n ], correlation coefficient matrix Corr 0 ∈ R n × n , and power ratio matrix Power 0 ∈ R n × n of all detected eye blinks, and the computed threshold delta) are obtained. After the calibration session, an EEG data streaming channel is established and a fixed-length time window is used to capture the streaming EEG data with a fixed stride. In the time step of t, the captured EEG data is to detect its local minimum with threshold delta, and then find its extreme points based on the local minimum (if detected). If the extreme points can be found, the signal segment between them sig t is obtained. With the Corr 0 from the calibration session, an extended correlation coefficient matrix Corr t can be computed by Eq. (2).
Similarly, an extended power ratio matrix Power t is obtained by Eq. (3).
After that, the concatenated patterns Sig t = [sig 0 sig t ] are clustered based on the feature. Corr t /Power t . Finally, the comparison of the clustering result and the result of the calibration session is done and the online detection result (idle, long blinks detected, or double blinks detected) can be obtained.

Augmented reality-enabled feedback interface
The above voluntary blink online detector can monitor the human intentional input. However, only two patterns (long blink and double blink) can be recognized which makes it almost impossible to accomplish a complex robot control purely using the limited types of EEG patterns. To this gap, we are inspired by the graphical user interface (GUI) of today's computer which presents users with suggestive or feedback visual contents on the monitor so that users can use the simple inputs (like clicking the mouse) to achieve the conveying of their intention. To transfer the interaction scenario from the 2D monitor to the 3D real world, we introduce a wearable augmented reality headset to provide humans some augmented visual feedbacks corresponding to their inputs, making the above-proposed brain-computer interface a closed-loop one. The rest of this section will present how to implement the four components of the proposed AR-enabled feedback interface: the control logic, the interactive path planning, and the command handling.

Control logic
In order to incorporate the eye blinks into the human-robot interaction, we design a control logic to process the human input. As shown in Fig. 8, the designed control logic is based on the finite state machine (FSM). In the FSM, four states are defined: (1) Standing-by (StdB), (2) Trajectory modification (TrajM), (3) Motion preview (MPrev), and (4) Command handling (CmdH); and in any state, the human can input two types of action, long blink (in solid arrow) and double blink (in dotted arrow), to control the transition of the current state. To facilitate memory, the long blink is defined as the action that can change the current state forward; and the double blink is defined as the action that can either change the current state backward (in state 1 and state 3) or perform operations within the current state (in state 2 and state 4).
In the StdB state, the robot stops and is awaiting the human's inputs. If the human performs a double blink, the robot would keep the original configuration and resume to execution state. If the human performs a long blink, the robot would enter the TrajM state and start listening to the interactive information conveyed by the human.
The combination of the TrajM state and the MPrev state could enable human to perform an interactive robotic path planning with a modify-and-preview process. In the TrajM state, the human could modify the robot's planning by visualizing and editing the trajectory using the BCI and the AR headset, which will be introduced in the following section. In the MPrev state, the modified planning would drive a motion simulation with a 3D virtual robot as a preview, which can give human an intuitive feedback corresponding to their intentional inputs.
After the interactive robotic path planning, the confirmupdated configuration will be used to generate the robotic executable command in the CmdH state after a long blink by human. And the robot executes the generated command after a double blink. During the robot operating, the human can impose an intervention to make robot into the StdB state.

AR-enabled interactive robotic path planning
In our previous work [12], we have developed an interactive path planning using hand gestures with an AR feedback. In this work, we combine the voluntary eye blinks and the head motion into AR-enabled interaction to achieve a hand-free robotic path planning. The last section has introduced how the voluntary eye blink changes the system state. This section will introduce the interactive method to perform path planning The AR-enabled interactive robotic path planning consists of (1) trajectory specification with head motion, and (2) ARbased motion preview. In the trajectory specification, the human's head motion is tracked and is mapped into the change of the trajectory of the robot's end-effector (EE). In the motion preview, a virtual robot is built in an AR scenario and simulated the motion with the modification trajectory.

Trajectory specification based on head motion capture
The trajectory specification is to offer the BCI-AR user an efficient way to target the object of robot, as well as to plan the end-effector (EE) trajectory on the fly. It includes three procedures: target specification, trajectory generation, and trajectory editing. As shown in Fig. 9, the workflow of three modules is controlled by the detected user eye blink signal.
Target specification The coordinate systems of the AR headset, robot, and the HRC workplace are shown in Fig. 10. In this setting, the robot's motion target is specified according to the instant pose of the human head, and is visualized through the AR headset.
At the time step t, the system enters the TrajeM state with a human head pose represented by a coordinate system H where a unit vector u ! t is defined to represent the head orientation and with a head-EE distance d t . The robot's target appointed by human can be initialized as P t by Eq. (4), where Reach is the space of the robot's reach, w P t = [ w x w y w z] and h P t = [ h x h y h z] are the coordinates of P t relative to coordinate system W and H respectively, and w T is the transformation matrix from coordinate system H to W which consists of a 3×3 rotation matrix w rot and a 3-dimensional translation vector w trans. Practically, the w T is obtained with PTC's Vuforia™ SDK [42] by dynamically tracking a 2D marker attached to the workplace.
Then at every updated moment t′, the AR headset can capture the motion of the human head represented by the transformation matrix of head motion ΔT which consists of a 3×3 rotation matrix Δrot and a 3-dimensional translation vector Δtrans. With the movement of human head, the appointed robot's motion target P t′ will change simultaneously based on Eq. (5) where the r T is the transformation matrix from coordinate system H to R, which consists of a 3×3 rotation matrix r rot and a 3-dimensional translation vector r trans. Since the robot is stationary relative to the workplace, the r T can be measured and calculated in advance. The P t′ will be fixed as the target after the BCI user performing a long blink.
Trajectory generation The position in joint space corresponding to the appointed target can be computed by the inverse kinematic solver. Interpolating evenly between the initial position and the computed target position, a discrete sequence of the joint's position can be obtained. With this sequence, a trajectory of EE can be generated by forward kinematic. A virtual curve and some spheres are rendered in the AR scenario to intuitively show the generated trajectory and its waypoints.
Trajectory editing After target specification and trajectory generation, the user also can edit the automatically generated trajectory. The user can move his/her head to focus a waypoint to be edited using head-gaze input, a built-in input model of HoloLens [43], then performs a double blink to confirm the edit point selection. The selected point moves in free space with head motion like the target specification module. Similarly, it will be fixed as a user-defined waypoint after the user performs a long blink. With the user-defined waypoint and two anchor points (the initial and the target points), a set of segmented trajectories can be computed through the trajectory generation module. By joining the Fig. 9 The workflow of trajectory specification segmented trajectories together, the edited trajectory can be obtained.

AR-based motion preview as feedback
The AR scenario for motion preview is built in Unity3D. As shown in Fig. 11, a virtual robot has been built in Unity3D which has the same size, shape, axis types, and hierarchy structure as the real robot ABB IRB1200. After the robot motion target was determined and the system entering the MPrev state, the determined P t′ is used to compute the robot's joints position q t′ by Fast IK [44] which is an open-source inverse kinematic solver specified for Unity3D. The computed q t′ then drives the built virtual robot built in the AR scenario to run a motion simulation as a preview corresponding to the specified robot configuration. The robotic motion preview animates in the developed AR scenario with a refresh rate f in frames per second (fps). In every frame of the AR scenario rendering, the refreshed joint position of the virtual robot q = [q 1 , q 2 , q 3 , q 4 , q 5 , q 6 ] is computed by Eq. (6): where q ;q 2 ;q 3 ;q 4 ;q 5 ;q 6 is the joints' speed consistent with real robot settings. At the same time as motion simulation, the EE's trajectory is also rendered through a visual curve and some key points.
According to the motion simulation, the human can either confirm the current configuration or re-plan the presenting trajectory. After confirming, the final configuration will be handled to generate the command for robot execution.

Command handling
The input to the command handling module is the trajectory of the robot's EE, namely the time series of the points of robotic planned paths, which is defined by human through the BCI-AR interface. This module is to translate the input information into the robotic executable command which can reconfigure the control program on the fly. To realize that, we designed a program for the robot controller with a structure as shown in Fig. 12. The program has three key variables which are the velocity of EE vel, point series of EE point, and the orientation of EE ori. The point is defined by human through the interactive path planning presented in the above section, which contains Cartesian coordinates of the path. The vel can be adjusted according to the safety status of the HRC workplace (e.g., vel reduce with the human-robot distance being closer [12]). The ori is defined by the task specification (e.g., in pick-and-place tasks, ori should be well-defined to achieve stable object grasping) which could be retrieved in a pre-defined knowledge library.
To make the I/O operation of the robot controller, we implemented this module based on the ABB RobotStudio [45] which can establish a connection to the controller and a developed C# software with ABB PC SDK [46] which can establish a connection with RobotStudio to read the robot running data and to modify the variable in the controller program.

Case study
In this section, we first evaluate the performance of the algorithm we proposed in Section 4 with the comparison with the related work; then we develop an industrial HRC case to show the effectiveness of the proposed BCI-AR interface with the comparison with a hand gesture-based AR interface. Fig. 10 The coordinate systems of the AR headset, robot, and HRC workplace

Performance of the voluntary eye blink detection algorithm
To verify the effectiveness of the proposed algorithm, we collected a dataset to test its performance of multiple voluntary blink detection. The dataset is collected from the same 12 subjects and in the same setup as the previous test presented in Section 4.1. The subjects acted with the guidance of a visual clue showing on the monitor while recording EEG data. As shown in Fig. 13, each subject was guided to perform 40 times long blink, double blinks, and idle status (the subject could perform normal blink or stay still).
The collected dataset with the size of 1,440 then is processed by the proposed algorithm implemented in Python. The detection result is presented as a confusion matrix shown in Fig. 14. The accuracy, precision, and sensitivity of the detection are defined by Eq.(7)~Eq. (9), where the TP and TF are the numbers of the true positive and true negative; the FP and FN are the numbers of the false positive and the false negative.
Based on that, the performance of the algorithm for each subject is also calculated, as shown in Table 1. From that, we found that an average accuracy of 94.31% was achieved. Among all subjects, the best performance occurred in the subject 4 and 12 with an accuracy of 99.17%, while the worst performance occurred in subject 8 with an accuracy of 82.33%. Among all classes, the double blink class holds the Fig. 11 The virtual model and the hierarchy structure of the robot Fig. 12 The structure of the robot program Fig. 13 The protocol of EEG data collection with the visual clue for each subject biggest precision and the idle class holds the biggest sensitivity.
In addition, we have done some performance comparisons with three related works. The first related work is based on dynamic positional warping (DPW) [27], which is in the template matching style. The second and third related algorithms are both based on supervised learning, which are based on SVM [28] and RBF [29] respectively. In this comparison experiment, we reused the 1,440-sized data set mentioned above. Due to that all the related work needs a training phase, we used 60% (864) of the data set for training and 40% (576) for testing. For the DPW algorithm, we set the length and the stride of the slide windows to 1000ms (50% of the signal length) and 100ms (5% of the signal length), respectively. For the SVM-based algorithm, we optimized the parameters (C and gamma) by comparing the performance of the different values of parameters. Similarly, for the RBF-based algorithm, we selected the optimal parameter (gamma) according to the same searching technique. The results are shown in Table 2. Our proposed algorithm has higher average accuracy and precision. Moreover, compared with the related work in Table 2, our proposed algorithm, which needs a calibration session of only 45 seconds, does not need any training phase or any labeled data.

Case study of an industrial HRC task
In this part, we develop a case study in an industrial HRC assembly task where human and robot, an ABB IRB1200, jointly perform a batch assembly task, as shown in Fig. 15, to demonstrate the effectiveness of the proposed closed-loop interface combining BCI and AR.  The assembly task consists of two parts to be assembled. Human could use the BCI-AR interface to program the robot motion trajectory on the fly to assign which part is the robot to approach and pick. The robot is mastered by the robot controller ABB IRC5 deploying ABB RAPID robot programming. The controller and a PC are connected via Ethernet by which the running data and the control command transfer. The PC deploys a robot control interface built upon the ABB PC SDK with C#, which can (1) monitor the robot status and modify its controller configuration, (2) establish the communication channel with the HoloLens via LAN, and (3) receive user input from the voluntary eye blink detector implemented by Python via TCP/IP protocol. The detector and the OpenBCI GUI are connected via the UDP protocol where an EEG data streaming is established and the online eye blink detection is realized. The AR headset achieved the tracking of a marker attached to the workplace to obtain the transformation between the AR headset and the workplace. Figures 16 and 17 show the snapshots of a user's point of view from HoloLens and the collected EEG data during a single interaction. At the moment t=0s, the user performed a long blink which starts the interaction. Then, a red sphere,  pointing out the robot's target, was rendered in the center of the user's perspective with the same depth as the distance between the user's head and robot's TCP, and it can be moved with the transformation of the user's perspective, as shown in Fig. 16(2) to Fig. 16(4). With a double blink performed at t=2.6s, the target was fixed. After a long blink performed at t=4.8s, a trajectory was computed and visualized as shown in Fig. 16 (5). The user moved his head and focused on a point in a visual trajectory. With a double blink at t=9.7s, the focused point turned blue and could be moved with the transformation of head pose, by which the previously computed trajectory could be edited, as shown in Fig. 16(6) to Fig. 16 (8). After a double blink at t=12.3s and a long blink at t=15.1 s, the modified trajectory and the motion simulation were rendered as a preview, as shown in Fig. 16 (9) to Fig. 16(12). After another long blink at t=22.3s, the robot executed the rewritten configuration as shown in Fig. 16 (13) to Fig. 16(16).
To better evaluate the performance of the presented system, we compared the performance of this system with [12] on some time indicators. In [12], as shown in Fig. 18, the input of the AR-enabled human-robot interaction is based on the hand gesture (HG) recognition of the HoloLens SDK and depends on some AR UIs. In the comparative experiment, 12 users performed the interactive process as shown in Fig.15 with two systems for 5 times each after the user learning sessions, where three indicators were recorded: (1) the total interaction time (TIT) which is defined as the time from the beginning of the interaction to the completion of the robot reconfigured motion, e.g., 29.5s in Fig. 16; (2) accumulative input time (AIT) which is defined as the accumulative time in the interactive path planning, e.g., 22.3s in Fig. 16; and (3) the average time per input (AT/I) which is defined as AIT divided by the number of user input, e.g., 22.3/7≈3.19s in Fig.  16. Figure 19 shows the performance comparison on the above indicators between the two systems. From that, we found that the proposed BCI-AR system has better performance on all indicators with the comparison of the HG-AR system (23.38% shorter in TIT, 25.19% shorter in AIT, and 13.83% shorter in AT/I).

Conclusion and future work
This paper, aiming to release the better potential of BCI and AR application in industrial HRC, proposed a closed-loop interface of human-robot interaction using the human eye blink input and the AR feedback. First, we proposed an EEG-based voluntary blink online detection algorithm which is in an automated unsupervised way and can achieve an average accuracy of 94.31% with stable performance across Fig. 17 The recording of EEG data and the result of eye blink detection Fig. 18 Examples of the user input in [12] users. Then, we designed an AR-enabled feedback interface that could switch the interaction state according to the different inputs and achieve an interactive robotic path planning through the AR-based modify-and-preview process. Finally, we integrated the proposed method into a case study in an HRC assembly task. The case study shows that compared with the hand gesture-based input method, the proposed eye blinkbased method can reduce the time of user input, which could improve the effectiveness of the communication between human and robot.
Towards autonomous remanufacturing, we will go deep into the complex operations of the disassembly process in remanufacturing (e.g., unscrewing, removal pins form holes) and explore more potential of the proposed system in our future research. Firstly, we will incorporate the output of disassembly sequence planning into the AR feedback to dynamically display the task-related information for the human worker to enhance their adaptation to the scenario in the collaborative disassembly tasks. Secondly, the AR-BCI control logic of the proposed system will be redesigned, where the planned disassembly sequence will be taken into account. Thirdly, an interactive robotic path planning suitable to the specified disassembly operations will be built, where the geometric features of different disassembly tools (e.g., unscrewdriver, gripper, etc.) will be incorporated into the planning phase. Fourthly, the proposed system could be used as an auxiliary channel for existing interaction solutions (e.g., gestures recognition, haptic interaction, speech processing), and its integration into the existing interaction system which could build up a more robust multi-modal human-robot interaction will be further studied.
Availability of data and materials The data and materials of this paper are available from the corresponding author on reasonable request.
Author contribution Zhenrui Ji designed and conducted the case study, analyzed the results, and wrote the paper. Quan Liu proposed the basic idea and contributed the materials. Wenjun Xu proposed the method design and experimental idea and also modified this paper. Bitao Yao proposed the idea of EEG signal processing. Jiayi Liu analyzed the data and conducted the case study. Zude Zhou modified the structure of the paper and contributed the method development.
Funding This research is supported by the National Natural Science Foundation of China (Grant No. 51775399) and the Fundamental Research Funds for the Central Universities (WUT: 2020III047).

Declarations
Ethical approval The subjects involved in this research are all volunteers, and the authors warrant that the paper fulfills the ethical standards of the journal. No conflict of interest exists in the submission of this paper, and the paper is approved by all authors for publication. This paper has not been published or presented elsewhere in part or entirety and is not submitted to another journal.
Consent to participate The subjects involved in this research are all volunteers; the authors warrant that the paper fulfills the ethical standards of the journal.
Consent to publish All authors are consent to publish this paper in the International Journal of Advanced Manufacturing Technology.

Competing interests
The authors declare no competing interests.