Bird’s -eye view provided in real-time via drone improves spatial perception for team sports

Sports-assisting technologies have been developed; however, most are to improve performances in individual sports such as ski, batting, and swimming. Few studies focused on team sports which require not only motor ability of individual players but also perceptual abilities to grasp positions of their own and others. In the present study, we aim to validate the feasibility of a visual feedback system for the improvement of space perception in relation to other persons that is necessary. Herein, the visual feedback system is composed of a flying drone that transmits the image to the participant’s smart glasses. With and without the system, the participant was able to see his/her own relative position in real time though the glass. Nine participants tried to position themselves on the line between two experimenters 30 m away from each other, which simulated the situation of a baseball cutoff man. As a result, the error distance between the participants’ position and the line significantly decreased when using the system than that without the system. Furthermore, after participants practiced the task with the system the error decreased compared to that before the practice. In conclusion, the real- time feedback system from the bird’s -eye view would work for improving the accuracy of space perception.

Assistive sports technologies can help users learn how to move their body by providing information about ideal and actual body movements [1][2][3][4] . Efficient learning methods may also help users by increasing motivation to engage in sports and other forms of exercise. Therefore, assistive sports technologies can contribute for players to maintain motivating for exercising. In various assistive sports technologies, real-time feedback systems with audio signals have been found to improve learning efficiency 56 . For example, Iwata developed a rhythmic auditory feedback system, in which auditory cues are linked with body motion, to improve baseball bat swing performance 7 . Their system helps players to improve their swing speed and improve their kinetic chain during upper body motion. Visual feedback systems have also been found to be useful for movement learning [8][9][10] . Rekimoto developed a sports assistant system that uses a drone camera for batting training 11 . The drone hovers behind a participant while they engage in batting and captures a visual image of the participant. Although the flying assistant system is able to give visual feedback regarding batting form to the batter, the impact of this feedback on performance is not yet clear.
So far, assistive sports technologies have mainly focused on improving performance in individual player sports. As far as we have investigated related works, few previous studies have examined the use of such systems in improving the movements of players in team sports. In team sports, all of the players on each team have the same goal, and each player must cooperate with the other players in accurate timing and spatial position. Therefore, it is important for players to quickly understand their position in the playing field relative to others.
In this study, we evaluated the feasibility of a visual feedback system for improving spatial perception in players engaged in team sports (Fig. 1). In this visual feedback system, a camera-equipped drone flew above players as they moved on a baseball field (Fig. 2). The drone transmitted an image of the field to transparent smart glasses worn by players, giving them a bird'seye view of the field in real time. In the experiment, we examined the movement of a baseball player who had the role of 'cutoff man', because the position of the 'cut-off man' in relation to other players is particularly important 1213 . We examined how far players could learn where they were located on the field based on the visual data provided by the drone through the comparison of players' motion among before, during and after the use of the system. The participants walked and stopped on the imaginary line connecting the two experimenters before, during and after using the system (Fig. 3). We validated two points: 1) Whether or not the player's self-localization improves when utilizing the visual feedback system compared to that before utilizing it, and 2) Whether or not effects on the player's self-localization lasted after exposure to the system. Finally, in the discussion, we discussed how effectively bird's eye view data worked for users to understand their self-position within an environment, including referring a cognitive feedback model to attribute our results to known theory.    Fig. 4 shows the relative error with respect to the target line in each participant. We evaluated improvements in spatial perception elicited by the real-time feedback system by measuring the distance error. Fig. 5 shows the (a) absolute and (b) relative error group data. The ANOVA for the absolute error data ( Fig. 5 (a)) indicated a significant main effect, as shown in TABLE I (F = 7.89, p = 0.00232). Where F means F-value which is decided by the Degree of freedom in each group on the Fdistribution, and p means the probability value. We used paired t-tests to compare the error in the trials before, during, and after the trials in which the participant received visual feedback regarding their position (p < 0.05), as shown in TABLEs II-IV. The error values in the visual feedback trials were lower than those before or after the feedback trials (p < 0.01, effect size  2 = 0.15,  2 = 0.14). The ANOVA for relative error data also showed a significant main effect, as shown in TABLE V (F = 3.54, p = 0.0450) and Fig. 5 (b). We used a paired t-test to compare the relative error before, during, and after visual feedback (p < 0.05), shown in TABLEs VI-VIII.

Discussion
Our results indicate that compared with the trials conducted before the feedback trials, the absolute error in spatial position was lower in the trials with visual feedback and the trials that took place after the visual feedback trials. Furthermore, most participants tended to stop before the line in all conditions (Fig. 5). Paired t-tests also showed a significant difference between the stop position in the trials conducted before the feedback condition and zero, which corresponds to perfect performance (p < 0.05). The results might have been caused by the wide viewing angle in the experiment. As humans usually have a viewing angle of more than 180°, the participant was able to look at the experimenter on each side. However, foveal vision was necessary to determine the correct position. Therefore, determining their own spatial position relative to the line between the two experimenters was not an easy task. In contrast, when they had access to the bird's-eye view from the drone, the participants could look at themselves from an objective viewing angle and confirm his/her own relative position. Therefore, the participant could easily recognize whether they had reached the line in the visual feedback trials. The participants learned from error using the visual feedback system. After the drone system trial, the participants would predict the self-position from the bird's eye view. In the participant's brain, not only the feedback but the feedforward loop would also work for prediction. One of the famous feedforward schemes is the feedback error learning model 14 . Fig. 6 shows the feedback error learning model. The feedback error learning model is the scheme that an inverse model produced based on a feedback loop works as a model for a feedforward loop. After the learning, not only the feedback but feedforward loop can also work. Fig. 6 (a) depicts natural visual feedback without the system. As noted above, this visual feedback is not necessarily accurate. Fig. 6 (b) illustrates the visual feedback given in the smart glasses system. The smart glasses feedback loop was added to the feedback provided by the natural movement of the eyes because the participant was able to see both their natural view and the drone view using the smart glasses. In this system, the participant was able to adjust the inverse model and learn about the error regarding their stopping point. In the trials conducted without the visual feedback system, the participants were not given error feedback and the starting potion was varied. Thus, the practice effect is not likely to explain the observed increase in task accuracy when they had access to the bird's-eye view image.   The results of this study by themselves are not necessarily sufficient to support the feedback error learning model. For example, in the detail of the inverse model box in Fig. 6, the values in the boxes corresponding to the feedback controller and the human body are unknown because we were not able to quantify the cognitive error. Brain activity measurement would be needed to quantify the amount of cognitive perceptual error. In future work, we hope to study the contents of these boxes via bio-signal analysis.
This study represents a pilot investigation of spatial perception related to sports. In this study, the participant moved but the experimenters did not. In future work, we hope to examine the use of this system for sports in which players must cooperate with other players who are always moving, such as soccer. The real-time visual feedback system could be very useful for situations where both the participants and experimenters are moving simultaneously. In addition, the system could help reduce errors in scoring during real game situations.
Further, AR/VR/mixed reality (MR) could be displayed on the smart glasses. With smart glasses that are transparent, like those used in the present study, the user could simultaneously look at the view from his/her own eyes, a third person or bird'seye view, and an AR/VR/MR display. However, such a display must be designed such that the user can intuitively comprehend complex information. AR/VR/MR applications to sports such as skiing 15 may enable players to confirm their score in real time while moving. Moreover, this system might enable the creation of new sports. Because the user can access an external viewing angle and their own natural viewpoint, they might be able to move differently. Such new sports might use technology in a way that is similar to e-Sports 16 or Cybathlon 17 .

Participants
Nine healthy adults (four men and five women with an average age of 21.5 and age range of 20-23 years) participated in the experiment. The participants wore a green cap to enable their position to be easily visible from above, smart glasses, a helmet, and a faceguard for safety. Informed consent was obtained from all participants before the experiment. The experiments were approved by the Waseda University Intuitional Review Board (IRB2014-287). All experiments were performed in accordance with relevant guidelines and regulations. An experimenter who controlled the drone had a Remote Pilot License. The experimenter is called "operator" in this paper.

Bird's-Eye View System
The image from the drone camera was shown on the smart glasses worn by the participant (Fig. 2). The smart glasses (Moverio, Epson Inc., Tokyo, Japan, weight 88 g, 0.42 wide panel, view angle 23.0) were transparent 18 so that the participants were able to see the environment through the glasses and the image from the drone camera simultaneously. Thus, they could judge their position from both first-person view and birds-eye-view information (Fig. 7). The image resolution was 960  540 pixels.
We used the Phantom 2 Vision+ drone manufactured by DJI, Inc. (China). The drone equipped a GNSS receiver and a pneumatic sensor so that it can position itself with 3 degree of freedom in global coordinates. Also, a dedicated console was prepared for users to control it as they like. In this experiment, the drone was controlled to be positioned above participants at a height of 40m from the ground. In addition to those, the drone equipped a 14.0-megapixel camera (DJI, 2018) and its realtime images can be transmitted to a smartphone running Android 4.0 (Galaxy, Samson Inc., Korea).
The image from the drone was transmitted to the Moverio smart glasses via the smartphone using Bluetooth at 15.0 fps. Latency, which was less than 200 ms when the drone was 40.0 m above the field and less than 130 ms when it was on the ground, was sufficiently small that the participants did not perceive a time lag 19 . Also, the field of view of the camera was 85.0, and is sufficient to capture all the field which would be used for this experiment.

Task
First, participants stood at the start position, and then walked quickly with a straight trajectory towards the imaginary line connecting two experimenters. The two experimenters stood 30 m apart and held pink umbrellas so that participants could see their position displayed on the smart glasses easily (Fig. 4). The experimenters were standstill during the experiment in order to validate spatial perception ability of the participants relative to the static positions of others. We measured an error as the distance between a participant's stop position and the target line. A third experimenter allocated participants to the initial position, which was randomly varied between 7.0-13.0 m from the target stop line to prevent participants from becoming familiar with the task. Participants were able to look at the positions of the experimenters. To validate feasibility, we selected a short and simple movement task so as to imitate reactions of "cut-off man" s, which require them to locate themselves in only a few seconds in a real game.

Experimental Condition
We conducted experiments with three conditions in the following order: 1) A participant completed the task without the smart glasses (we call this "before feedback"); 2) the participant wore the smart glasses and completed the task with the birds-eye view image from the drone camera (we call this "feedback"); and 3) as in the first condition, the participant completed the task without the smart glasses (we call this "after feedback"). We evaluated the effect of learning by comparing the results of the "after feedback" trials to those of the "before feedback". There were 6 different start positions that ranged from 7.0-13.0 m with 1 m pitch from the target line. The 6 start positions were randomized for each of three conditions, with three trials for each start position. Therefore, each participant carried out the task 54 times in total. We did not give feedback to participants regarding error values in each trial.

Analysis
We used images obtained from the drone camera to measure a participant stop position. Based on colors of a cap and umbrellas, we extracted 1 green point and 2 pink points representing the position of a participant and that of the experimenters respectively from an image. We estimated the distance error according to the number of meters per one pixel using the distance between the two experimenters (30.0 m) positioned against the long side of the image vertically. The analysis was conducted as follows.

1) Capturing.
The operator adjusted the position of the drone with a display indicator so that the two experimenters with pink umbrellas were captured in the middle region of the camera lens and oriented vertically.
2) Image correction. The lens distortion in the drone camera image was corrected using Adobe Premier Pro and the DJI Phantom Vision 2+ Preset tool using the field of view (85.0). Camera image needs to be pre-processed because of distortion on camera lens. The images taken by the drone camera was corrected under the assumption that the distortion can be considered as barrel distortion, which is typical distortion on low magnification camera lens.

3) Extraction.
Images of the trials were extracted from the video after each experimental session.

4) Calculation of pixel per meter.
We recorded the number of pixels between the two umbrellas using Adobe Illustrator. This number of pixels corresponds to 30 m in the real world. We calculated pixels per meter as a reference based on the number of pixels between the two experimenters.

5) Measurement on pixel.
We then drew a perpendicular line from the participant to the line between the two experimenters. We recorded the number of pixels that made up this perpendicular line. 6) Measurement on meter.