2.1 System Architecture
The overall structure of the system is as shown in Fig. 1. Electrocardiograms (ECGs) and electroencephalograms (EEGs) are collected from the sensors attached to the passenger. The data are then integrated by the biosignal device management system and delivered to the biosignal monitoring system. Additionally, the passenger’s gaze position and head angle are detected by the gaze sensor installed on the joystick and camera, respectively. These data are visualized together in the monitoring system and allow for the remote control in the event of sensor malfunction.
2.2 Face Tracking Module
As shown in Fig. 1, the M1045-LW IP camera by Axis Communications is used to establish the image streaming function for the pilot. As the system provides 1080p resolution and supports real-time streaming protocol, the video can be streamed in real-time on a remote PC as long as it is connected to the same network.
After the image is streamed to the biosignal monitoring system, the face detection and tracking algorithm by MediaPipe is used to recognize the region of interest on the face, and the facial tracking algorithm is used to track 468 three-dimensional facial landmarks and calculate the head angle.
2.3 ECG Monitoring Module
Figure 3 shows the equipment used to acquire the ECG data. The data collected with the ECG sensor attached to the user’s arm, as seen in Fig. 3(c), are transmitted using a wireless wearable biosignal transmitter, such as the BioNomadix Transmitter in Fig. 3(b). The data are transmitted to the biosignal data collection system (BIOPAC MP160) in Fig. 3(a) and then to the biosignal monitoring system to be visualized, as shown in Fig. 4.
2.4 EEG Monitoring Module
Figure 5 displays the device for the pilot’s EEG measurement. EPOC Flex by EMOTIV collects EEG data through up to 32 channels. In this system, 16 EEG channels (AF3, AF4, F3, F4, F7, F8, FC5, FC6, T7, T8, TP9, TP10, P7, P8, O1, and O2) were used. The collected data were visualized through the monitoring system shown in Fig. 6. Figure 6(a) represents the sensor’s contact quality for each channel with colors—green (good), orange (normal), red (poor), and black (no contact). The data corresponding to each channel are visualized in Fig. 6(b).
2.5 Gaze Tracking Module
Figure 7 shows the sensor (Eye Tracker 4C manufactured by Tobii) that tracks the pilot’s gaze. The sensor is located on the back side of the joystick, as shown in Fig. 7(b). The sensor uses two eye angles to measure the screen coordinates of the pilot’s gaze. The calculated coordinates are transmitted to the monitoring system and visualized in the form given in Fig. 8.