Robotic Arm Control System Based on Brain-Muscle Mixed Signals

 Abstract: Aiming at the existing problems of BCI (brain computer interface), such as single input signal source, low accuracy of feature recognition, and less output control instructions, this paper proposes a robotic arm control system based on EEG (electroencephalogram) and EMG (electromyogram) mixed signals. The system flow is as follows: Firstly, the EMG signal of the unilateral arm and the EEG signal of the left and right hand motor imagery is collected synchronously. Then the collected EEG and EMG signals are extracted and classified, and the corresponding classification instructions are obtained. Finally, the multi-instruction real-time control of the robotic arm is realized under the classification instruction. The experimental verification results show that: The 10 subjects all realized the real-time multi-command control of the robotic arm, and the average recognition accuracy of each action reached more than 94%. The proposed system enriches the diversity of hybrid BCI and provides a theoretical basis and application foundation for the extended application of BCI in robotic arm control.


Introduction
Intelligent human-computer interaction is a high-end human-computer interaction technology based on the detection of the human bioelectrical signal, with feature engineering and machine learning as the core, to realize the machine's active understanding of human behavior intention, it represents the latest hot spot and trend of human-computer interaction [1]. In recent years, human-computer interaction  Li-Wei Cheng clw1016@sina.com technology based on EEG or EMG [2] has achieved significant application achievements in rehabilitation medical treatment [3], gait recognition [4], human exoskeleton [5], and other fields. However, the majority of intelligent human-computer interaction systems related to BCI or MCI (muscle machine interface) adopt a single mode of bioelectrical signal to control external devices. This single-mode has the disadvantages of low information transmission rate, single information source, and less output control instructions of external equipment. Compared with the single-mode BCI system, the multi-mode hybrid BCI [6] can effectively make use of the fusion and complementarity of multi-source information, and make up for the shortcomings of the existing BCI in low recognition accuracy and fewer control instructions.
At present, many scholars have carried out in-depth research on the hybrid BCI with multi-source information and achieved remarkable achievements. Li [7] proposed a multifunctional prosthetic control method based on EEG and EMG fusion to achieve high-precision identification of upper limb movement, aiming at the problem of too few residual muscles and insufficient EMG signal source of amputees. Duan et al. [8] implemented a multimodal online BCI system that combined μ rhythm, SSVEP, and MI-EEG (motor imagery electroencephalography) signal to control the robot to grasp objects. Ma et al. [9] proposed a multimode cursor control system based on the combination of MI (motor imagery) and mVEP (motor onset visual evoked potential), which realized the overall improvement of the 1 School of Modern Post(School of Automation), Beijing University of Posts and Telecommunications, Beijing 100876, China 2 Beijing Aerospace Measurement & Control Technology Co.，Ltd., Beijing, 100041, China control efficiency of the BCI system. Luis Mercado et al. [10] combined EMG and EEG to simulate tibial-femoral joint motion control, the simulation of the user's motion intention and dynamic behavior is realized. Heba Ibrahim Aly et al. [11] proposed a new hybrid BCI model that combined EEG and EMG signals to effectively improve limb movement control in amputees above the elbow joint. Koushik Bakshi et al. [12] proposed a hybrid strategy of EEG and EMG, which realized the motion control of the elbow, wrist, and finger. Aya Rezeika et al. [13] proposed a spelling controller that combines SSVEP and EMG signals to achieve faster letter spelling performance without compromising accuracy. Andrea Sarasola-Sanz et al. [14] proposed a control system based on hybrid brain-machine interfaces (EEG and EMG) to achieve real-time control of the 7-DOF (degree of freedom) exoskeleton of the upper limb. Tomasz Kocejko [15] proposed a control system combining EMG and EEG and added the detection and analysis data of the eye tracker to realize the movement detection and analysis of prosthetic limbs. Jingsheng Tang et al. [16] proposed a robotic arm control method combining EMG and EEG, using leg EMG signal to quickly and reliably select the currently activated joint, and using motor imagery signal to accurately control the computer interface. Agata Manolova et al. [17] proposed a multimodal fusion method based on user electromyography, which achieved more reliable control on the precondition of reducing fatigue and inattention during use. Ludovico Minati et al. [18] achieved the successful control of a 5+1 DOF robotic arm by simultaneously collecting bioelectrical signals such as eye electricity, jaw electromyography, brain electricity, and head movement through wearable devices.
From the above studies, it can be seen that multi-source information mixing improves the output accuracy of the BCI system by using the mixed enhancement of two or more different bioelectrical signals. However, this multi-mode fusion requires the subjects to complete multiple tasks at the same time with higher attention, and too long task time will cause excessive fatigue of the subjects, affect the stability of the system control, and the output category of the device control instructions is very limited. To improve the control stability of the BCI and provide multiple control commands, this study design a robotic arm control system based on EEG and EMG mixed signals based on the idea of a hybrid BCI. Firstly, EEG and EMG signals are collected synchronously. Then the EEG and EMG signals are featured extraction and classified by parallel mode, and the corresponding classification instructions are obtained. Finally, the multiinstruction real-time control of the 6-DOF robotic arm is realized under the classification instruction.

Brain-muscle Mixed Signals Robotic Arm Control System
The block diagram of the robotic arm control system based on EEG and EMG mixed signals is shown in Figure 1. The system is mainly divided into three parts: Signal generation and acquisition, signal processing, and robotic arm task flow. Firstly, the corresponding experimental paradigm is designed for the task of the robotic arm, and the EEG and EMG signals are collected to reflect the movement state of the subjects. Then, the EEG and EMG signals are preprocessed, feature extraction, classification, and recognition respectively, and the EEG and EMG control instruction categories are obtained. Finally, the real-time control of the robotic arm is realized based on the corresponding EEG and EMG control instructions.

Figure 1
Block diagram of the robotic arm control system with EEG and EMG mixed signals 3 Materials and Algorithms

Signal Generation and Acquisition
According to the task requirements of the robotic arm, the experimental paradigm of EEG and EMG is designed. According to the prompts of the experimental paradigm on the screen, the subjects performed corresponding gestures and left and right hand motor imagery, and then the corresponding EMG and EEG signals are obtained. EMG signal acquisition: According to the distribution of human arm muscle groups, the EMG electrode adopts the ring attachment method with a total of 10 channels. The electrode distribution is shown in Fig. 2 (a). Gesture movements include wrist introversion, wrist extroversion, fist clenching, hand opening, finger kneading, and rest, as shown in Fig. 2 (b). The EMG training is performed in single trials, and the training duration of each movement is 2s, with a total of 12s for 6 movements.
EEG signal acquisition: According to the international standard 10-20 system [19], EEG signals of four channels C3, C4, FZ, and A1 are collected by the EEG electrode. A1 left mastoid is selected as the reference electrode, and FZ forehead as the central grounding, as shown in Fig. 2(c). There are 10 trials for left and right hand motor imagery training. The single-trial lasted for 2s, in which the left and right hand motor imagery for 1s each. Neuroscan EEG acquisition system is used to acquire EEG and EMG signals synchronously. The sampling frequency is 500Hz, and the signals are processed by a 50Hz power frequency notch. Fig. 2(d) is the schematic diagram of the timing sequence of the EMG experiment. The experimental procedures are as follows: Step 1: When t=0~2s, a cross cursor appears in the center of the screen, indicating the START state, indicating that the subject is about to START the EMG training, and he/she needs to pay attention and wait for the action prompt instruction.
Step 2: When t=2~14s, on the screen, there are six sports pictures of unilateral wrist introversion, wrist extroversion, fist clenching, hand opening, finger kneading, and rest. The subjects performed the corresponding arm movements according to the prompts. The six movements are performed in sequence, and each movement is completed in 2s Step 3: When t=14~16s, a cross cursor appeared on the screen, indicating that the subject is about to start the EEG training and needed to pay attention and wait for the action prompt instruction.
Step 4: When t=16~36s, the left, and right indicator arrows appeared on the screen. According to the direction of the arrows, the subjects performed the corresponding left and right hand motor imagery. The left and right hand motor imagery tasks are performed alternately. The left and right hand motor imagery are trained 10 times, and the training time is 20 seconds.
Step 5: After the EEG training, the screen is in the END state, indicating the END of the training experiment.

EMG Signal Processing
(1) EMG signal preprocessing 12 12 ( ) [ ( ), ( ),..., ( )] where N is the total number of sample points, n is the number of EMG channels, m is the number of sampling points. During the acquisition process, the EMG signal is subject to various interferences, resulting in a large number of artifacts. To improve the quality and decoding effect of the EMG signal, it is necessary to preprocess the collected EMG signal. The preprocessing process is shown in Figure 3. As can be seen from the figure, the preprocessing [20] of EMG signal includes DC component removal, filtering, artifact removal, baseline correction, denoising, etc. The high efficiency of signal processing can be realized by adding several manual checks in the preprocessing process. The preprocessed signal is Aiming at the motion range of each joint in the exoskeleton, considering the lossless and real-time characteristics of feature vectors in the time-frequency domain, and the relatively simple extraction, a feature subset extraction method in time-frequency domain is proposed.
To ensure the continuity of features, the method of setting time window and incremental window [21] is adopted to extract features (as shown in Figure 4).
MAV reflects the total electromyography. b) Root Mean Square [23], RMS RMS reflects the effective amplitude of the EMG signal at a certain time of exercise. c) Zero Crossings [24], ZC  is introduced to prevent small-value noise. ZC represents the number of sign change in signal amplitude per unit time (usually 1s), and ZC can reflect signal frequency. d) Slope Sign Changes [25], SSC x t x t (4)  is introduced to reduce the interference caused by noise on the sign change of slope, the SSC can roughly reflect the frequency of the signal. e) Waveform Length [26], WL x t x t WL (5) WL reflects the waveform complexity of the EMG signal. f) Willison Amplitude [27], WA WA reflects the level of muscle contraction. g) Variance [28], VAR VAR reflects the power of the EMG signal. h) Log Detector [29], LogD LogD reflects the external force assessment of the muscle. i) Auto Regression Coefficient [30], ARC Where ( ), 1,...,  a k k p is the AR model coefficient, namely the extracted feature, p is the model order, and () et is the model white noise. ARC describes each EMG signal value as a linear autoregressive time series, reflecting information about the state of muscle contraction. j) Cepstrum coefficients [31], Ceps a l a j c l j l (10) Where () al is calculated from ( ), 1,..., ,...,  a k k l p in Equation (9), Ceps reflects the change rate information of signal in different spectral segments.
k) Median frequency [32], l) Mean frequency [33], ME f The definitions of    P and  are the same as above and ME f reflect the degree of muscle fatigue. The selection of features directly affects the result of motion recognition. A good feature should show the consistency of parameters in the same motion mode, and at the same time, it should show the obvious difference of parameters in different motion modes. Feature selection is essentially an optimization problem. Feature selection is essentially an optimization problem. M features are selected from the original N features to form a feature subset, among which M meets  MN , and the subset is optimal when evaluated by a certain evaluation standard. To optimize feature selection, this system takes the average value of the extracted feature mean and standard deviation of the mean [34]: Where n T is the normalized mean of the characteristic mean of the data in the group n , T is the mean of n T , N is the number of data groups calculated in the experiment, and S is the standard deviation of n T . FBP can be used to measure the quality of features. The characteristics of different features can be obtained by the characteristic average value and FBP value.

(3) Recognition and classification of EMG signal
The neural network [35] is used to train the feature subset. A neural network consists of an input layer, hidden layer, And then train X to get optimal opt X , so that

EEG Signal Processing
(1) EEG signal preprocessing x t x t x t x t R is the collected EEG signal, where N is the total number of sample points, n is the number of EEG channels, m is the number of sampling points. During the acquisition process, the EEG signal is subject to various interferences, resulting in a large number of artifacts. To improve the quality and decoding effect of EEG signals, it is necessary to preprocess the collected EEG signal. The preprocessing process is shown in Figure 5. As can be seen from the figure, the preprocessing [36] of EEG signal includes DC component removal, filtering, artifact removal, baseline correction, remove of the interference, etc. The high efficiency of signal processing can be realized by adding several manual checks in the preprocessing process. The preprocessed signal is

Figure 5 EMG signal preprocessing process
(2) Feature extraction of EEG signal A feature extraction method based on energy (secondorder moment [37]) and wavelet packet transform is proposed by considering the relevant information of EEG between different channels and combining the comprehensive characteristics of signal energy and timefrequency domain. 1) Energy (second-order moment) EEG signal is collected by EEG cap and expressed in the form of amplitude voltage, the instantaneous energy is denoted as: According to equation (17), the average energy of each lead signal is calculated respectively, and the signals with obvious differences in periods are selected for feature extraction.
2) Wavelet packet transform Wavelet packet [38] decomposition is performed for the 0 g is a low-pass filter, m is the sequence number of filter coefficients, r is the sequence number of wavelet packet subspace, and 1 g is a high-pass filter. After wavelet packet decomposition, the original signal is divided into several wavelet packet subspaces according to frequency bands. The frequency bands corresponding to each subspace of layer l are respectively: (3) Recognition and classification of EEG signal Support Vector Machine (SVM) [39] is a machine learning method based on statistical learning theory. For a binary classification problem, SVM has stronger adaptability, better classification ability, and higher computational efficiency. Common kernel functions of the SVM method include linear kernel, polynomial kernel, and radial basis kernel, etc. Different kernel functions can be used to construct different SVM classifiers. In this study, the radial basis kernel function is used: x represents the input sample i , and  is the kernel parameter. For SVM, kernel parameter  and error penalty factor C are the main parameters that affect the performance. The parameter  affects the data distribution after spatial transformation, while the parameter C determines the convergence rate and generalization ability of SVM. In this study, the grid search method is used to determine parameters  and C , and a 10-fold crossvalidation method is used for verification. That is, of the 200 times of data, 180 times are known whether the experiment is to the left hand motor imagery or the right hand motor imagery, which is used as the training set, and the other 20 times do not give the motion mode, which is used as the test set.

Robotic Arm Task Flow
The Robotic arm used in the brain-muscle mixed signals robotic arm control system is AUBO-I5 6-axis robotic arm from AUBO (Beijing) Robotics Technology Co., Ltd. According to the working tasks of the robotic arm, the overall movement process of the robotic arm is as follows: Firstly, the robotic arm is initialized. Then through four movement steps (position 1, position 2, position 3, position 4), the end of the robotic arm reaches the specified position, ready to grasp the target workpiece, after grabbing the target workpiece, through three steps (position 3, position 2, position 1) to reach the position where the workpiece needs to be placed, ready to place the target workpiece. Finally, after the target workpiece is placed, the robotic arm returns to the initial position. The action sequence of the robotic arm is shown in Table 1.

Experimental Verification and Result Analysis
To verify the effectiveness of the brain-muscle mixed signals robotic arm control system, 10 subjects in the syndrome set were tested and verified. The subjects ranged in age from 22 to 33 years old and had no medical history. Informed consent was signed with each subject before the experiment. The experimental environment was quiet without absolute noise. The subjects sat in a comfortable seat with a horizontal distance of 70~80cm from the computer screen. To ensure the quality of data collection, alcohol cotton was used to clean the muscle and skin surface of the subjects before the experiment, and conductive paste was used to reduce the impedance between arm skin, scalp, and electrode. Firstly, each subject was informed of the procedure of the experimental paradigm. After all the electrodes were placed, the right hand was placed naturally on the table. Then, according to the prompts on the computer screen, the subjects performed the corresponding unilateral arm movements and left and right hand motor imagery training, and generated the training template. Among them, one unilateral arm six movements, and 10 left and right hand motor imagery was required. Finally, the subject can control the robotic arm in real-time. Experimental verification is shown in Figure 6.

Figure 6
Brain-muscle mixed signals robotic arm control experiment

Signal Processing Result
The EEG and EMG signals of the subjects were processed according to the algorithm flow in section 2.2. Now the signal processing flow of subject 1 is selected for detailed analysis.

EMG Signal Processing
(1) EMG signal preprocessing The effective frequency components of the EMG signal is distributed in the range of 0~500Hz, and the main energy is concentrated in the range of 10~200Hz, therefore, 10~200Hz bandpass filtering was performed on the original EMG signal, after removing the DC component, filtering, artifact removal, baseline correction and denoising, various noise interference in EMG signal can be effectively removed. Results of EMG signals of 6 gestures after preprocessing are shown in Figure 7.

Figure 7
Preprocessing results of EMG signals of arm movements (2) Feature extraction of EMG signal The pretreated EMG signal was extracted and the FBP values of 5 non-rest movements (T1: Wrist Introversion, T2: Wrist Extroversion, T3: Fist Clenching, T4: Hand Opening, T5: Finger Kneading) were calculated, as shown in Table 2. The smaller the FBP value is, the smaller the influence of data group change on the change of feature is, and this feature has better consistency to the motion. The four characteristics of the EMG signal: MAV, ZC, ARC, and fMD, are selected as the feature vectors of the EMG signal by integrating the types of features and FBP values.

(3) Recognition and classification of EMG signal
The four kinds of EMG signal features were imported into three commonly used classifiers, namely LDA (linear discriminant analysis [40]), SVM, and NN (neural network), respectively, and the classification results obtained are shown in Table 3. As can be seen from Table 3, the LDA method can identify EMG movements, but the recognition accuracy is relatively low, indicating that EMG feature samples are linearly inseparable. The overall recognition effect of the SVM method is better than that of LDA, and the highest recognition rate reaches 95.29%. The recognition rate based on NN is above 90%, and the highest recognition rate is 97.01%. The overall effect is better than that of LDA and SVM. removal, baseline correction, interference removal, and other operations were carried out. According to the time sequence of the EEG experimental paradigm, the EEG data of C3 and C4 were segmented once every 1000ms, and a total of 10 groups of data were segmented. The results of EEG signals preprocessing are shown in Figure 8.

Figure 8
Results of preprocessing of C3 and C4 EEG signals (2) Feature extraction of EEG signal Energy (second-order moment) and wavelet packet decomposition and reconstruction were used to extract the low-frequency average power difference between C3 and C4 of left and right hand motor imagery EEG signals, which were used to distinguish the EEG characteristics of left and right hand motor imagery. The experimental data were segmented 10 times, 10 groups of left hand motor imagery and 10 groups of right hand motor imagery. That is, in C3/C4, the left hand and the right hand were 10 times each. The average power spectrum of the motor imagery of the left and right hands is shown in Figure 9 and Figure 10. As can be seen from the figure, there is a significant difference in average power between 0 and 1000ms. Therefore, an EEG signal within 0~1000ms is selected for wavelet packet feature extraction.  The average power of the EEG signal was decomposed by wavelet, and the low-frequency information in the fifth layer (0~15.625Hz) was reconstructed by wavelet coefficients.
Wavelet coefficients were used to reconstruct the fifth layer reconstruction information of the difference between C3 and C4 average power spectra, and power scale features of low-frequency signals were extracted. Figure 11 and Figure 12 shows five groups of left hand motor imagery features and right hand motor imagery features of the fifth layer wavelet reconstruction. Compared with the subjects' left and right hand motor imagery, the five groups of features have certain differences, which can be used for the recognition and classification of left and right hand EEG features.

Figure 11
Characteristics of left hand motor imagery The EEG signal features were imported into three commonly used recognition classifiers, namely LDA, NN, and SVM, respectively, and the classification results obtained are shown in Table 4. As can be seen from Table 4, the LDA method can identify EEG movements, but the recognition accuracy is relatively low. The overall recognition effect of the NN method is better than that of LDA, and the highest recognition rate reaches 85.53%. The recognition rate based on SVM is above 90%, and the highest recognition rate is 91.24%. The overall effect is better than that of LDA and NN.

Overall Experimental Results
For the robotic arm control system based on the mixed brain-muscle signals proposed in this paper, the experimental verification was carried out on 10 subjects. After each subject was familiar with the instructions and preliminary training, 20 groups of experimental operations were carried out respectively, and a total of 200 times of data statistics were conducted. The experimental results are shown in Table 5, showing the total number of instructions, the number of wrong instructions, and the corresponding accuracy rate of the 10 subjects in 7 task commands of wrist introversion, wrist extroversion, fist clenching, hand opening, finger kneading, left hand motor imagery and right hand motor imagery. As can be seen from Table 5, in the experimental data results of each subject, the number of wrong instructions of each subject is less than 3 (including 3), and the accuracy of each instruction of other subjects is higher than 90% (including 90%) except the accuracy of hand opening instruction of S8 of the subject is less than 90% (the accuracy is 85%). In terms of the overall experimental data results, except the accuracy of the right hand motor imagery instruction is lower than 95% (the accuracy is 94%), the accuracy of all other instructions is higher than 95% (including 95%).

Conclusions
In this paper, based on the idea of a hybrid BCI, a hybrid EEG and EMG signals control system for the robotic arm is proposed, which combines unilateral arm and left and right hand motor imagery. To realize this system, the corresponding experimental paradigm of gesture and motor imagery is designed according to the task requirements of the robotic arm, the hand gesture is set as the motion position instruction of the robotic arm, and the left and right hand motor imagery is set as the end grasp instruction. To face the practical application, the EEG and EMG signals are processed by synchronous mode, and according to the characteristic of the EEG and EMG signals, different algorithm flow is adopted. To ensure the safety and reliability of the system, the kneading action of the finger kneading is set as the EEG mode switching instruction. The experimental results show that the average accuracy of robotic arm control command can reach 94%, and the subjects have no obvious fatigue phenomenon when performing the operation task. The proposed control system not only enriches the diversity of hybrid BCI but also provides a practical and theoretical basis for BCI technology in the field of the robotic arm. Figure 1 Block diagram of the robotic arm control system with EEG and EMG mixed signals EEG and EMG signals acquisition and experimental sequence diagram Brain-muscle mixed signals robotic arm control experiment Figure 7 Preprocessing results of EMG signals of arm movements Figure 8 Results of preprocessing of C3 and C4 EEG signals Average power of C3 and C4 left hands Figure 10 Average power of C3 and C4 right hands Figure 11 Characteristics of left hand motor imagery Figure 12 Characteristics of right hand motor imagery