A study of Human pupil orbit model estimation method for the eye-gaze tracking.

Background: Eye tracking is a technology that detects the direction of the user's gaze in real time. This technology is used in various fields such as analyzing user emotions and behaviors by analyzing the length of time the gaze stays in a specific direction or the path of movement. Recently, Eye-gaze tracking is a popular to a non-contact interaction technique for the minimization of covid-19 infection. A study conducted in the past have a limitation in that the ocular rotational center point (ORCP) cannot be found in common. For this reason, each research is continuously researching to overcome this problem by using various sensors, structures, and phenomena. In spite of, a tiny movement of the subject cause an error in accuracy during the process of Eye-gaze tracking. In this paper, we propose a novel method for Eye-gaze tracking under the condition of photographing the front of the face using a camera. We named this position set as Human pupil orbit model (HPOM) and it can express as a spherical equation because it looks like a sphere in 3-dimensional coordinate. Results: A simulation experiment was performed to confirm the performance of the proposed method. The error of HPOM estimation appears under 1 pixel in the first experiment both ORCP and a radius. In addition, it was confirmed that the process of estimating HPOM can be conducted as a real time in the second experiment. Conclusions: The proposed method was able to overcome the common problem of existing studies that could not find the rotational center of an eye-ball. In addition, HPOM may be secured in real time even if the subject moves.


Introduction
Eye tracking is a technology that detects the direction of the user's gaze in real time. This technology is used in various fields such as analyzing user emotions and behaviors by analyzing the length of time the gaze stays in a specific direction or the path of movement [1,2]. In addition, it is applied to daily life, such as configuring a Smart Home environment by combining a Human Interface Device and IoT technology to control various devices such as PCs and smart TVs [3,4]. Recently, Eye-gaze tracking is a popular to a non-contact interaction technique for the minimization of covid-19 infection.
There are two types of Eye-gaze tracking. One is using electrodes and another one is using a camera. The method of using electrodes tracks the gaze by attaching electrodes around the eyes and analyzing the signals generated by eye movements [5]. The method of using a camera is performed by photographing close-up eyes or photographing the front of the face and analyzing it. As methods of tracking the gaze close to the eye, a method using a smart lens that emits a laser and a method using the position of a glint created in the eye by attaching a near-infrared light source to glasses have been reported [6,7]. The method of photographing the front of the face is being studied as a method of tracking the gaze by analyzing the shape of the pupil or iris through an image processing technique after photographing the face [8][9][10]. Further, research has been conducted on a method of tracking the gaze using a glint that reflects a light source on the iris [11][12][13][14]. A study conducted in the past have a limitation in that the ocular rotational center point (ORCP) cannot be found in common. For this reason, each research is continuously researching to overcome this problem by using various sensors, structures, and phenomena. In spite of, a tiny movement of the subject cause an error in accuracy during the process of Eye-gaze tracking.
In this paper, we propose a novel method for Eye-gaze tracking under the condition of photographing the front of the face using a camera. The novel method is a acquiring a position which the pupil can move in 3-dimensional coordinate previous of Eye gaze tracking as a preprocessing. We named this position set as Human pupil orbit model (HPOM) and it can express as a spherical equation because it looks like a sphere in 3-dimensional coordinate. Although HPOM and an eye-ball model are represented as a sphere, the information they represent is different. If Eye-ball rotating in place, HPOM appears equally as Eye-ball. But, it is well-known that Eye-ball doesn't rotating in place when gaze direction is change because of Eye-ball center is different from ORCP. The proposed method is consisting of step by step including theory that from analysis of a pupil shape to calculating of HPOM. After HPOM was obtained through a whole process, Eye gaze tracking is can conducting in real-time using a 2-dimensional positions of the pupil center point (PCP). This method is can estimate HPOM rapidly and stably because of it consuming a few in number data even though the subject is move or change.

Experiments
The purpose of the experiment was to verify the validity of the theory proposed in this paper and to analyze the influence of variables in the fitting process. For the experiment, the x-axis, y-axis, and z-axis were all 500 pixels in size and a space with a unit interval of 1 pixel was set. The position of the center point in the space was 250 pixels on the x-axis, y-axis, and z-axis. An HPOM with a radius of 250 was constructed. A total of 100 samples were obtained by converting the random position coordinates on the HPOM surface to positions of x and y axes and RSA. In addition, a dataset with 200 TSs with an inlier and outlier ratio of 1:1 by randomly generating 100 noise data was constructed as shown in Fig. 11. (a). Fig. 11. (b) shows the distance on the xy-plane of the dataset and the distribution of RSA based on the center of the HPOM.
In order to analyze the performance according to independent variables the number of ESs, the value of AAE and MDG, the experiment was performed by setting the processing time, the location and distribution of the average ORCP, the radius of the HPOM, and the max cost as the dependent variables. The experiment was carried out in two sessions. The first experiment was repeated 200 times while gradually increasing the number of ESs from 3 to 14 in order to check the accuracy of the HPOM estimated according to change in the number of ESs. In order not to affect the first experiment, AAE was fixed at ±5 degrees and MDG was fixed at ±0 degree. Results were analyzed by performing a total of five experiments.
In the second experiment, the number of ESs was fixed by reflecting results of the first experiment to confirm the performance according to changes in AAE and MDG. The MDG was then continuously increased by 1 degree from ±0 to ±5 degrees and repeated 200 times. AAE increased MDG. It was fixed constantly during the experiment.
Results were analyzed by performing the experiment 30 times, changing by 1 degree from ±1 degree to ±5 degrees

Results
The experiment was conducted using Visual studio 2017 and Open Source Computer Vision, an image processing library, on a PC with Intel Core i7-7700k CPU and 64G RAM to configure the program and conduct the experiment. Open Source Computer Vision includes various image processing functions and provides functions specialized for matrix operation. HPOM could not be secured when three were ESs during the experiment. Therefore, the results were analyzed by excluding when ES was three. As the number of ESs increased, the processing time gradually increased. In addition, it did not affect the average cost. The estimated ORCP is constant regardless of the number of ESs. In Table 1, the distribution of ORCP that displayed as a distance is a tendency to decrease follow ES increases, but when it exceeds a certain number, the processing time and the distribution increase.
The second experiment was performed by the number of ESs to be 7 that showed the average performance in the first experiment. Table 2 shows the second experiment result. The processing time was consumed over 1 minute when MDG was at 0 degree, and less than 1 second when MDG was above 1 degree. Furthermore, ORCP and Radius of HPOM was constant value averagely on a 1 pixel range. The distribution of ORCP is most concentrated when MDG is 0 independently of AAE. However, it was irregular when MDG is over 0. The average cost is directly proportional to AAE without being affected by changes in MDG.

Discussion
We confirmed that HPOM can be estimated according to the proposed method through the first experiment and it can be performed in real time through the second experiment. We were analyzed the result of our research without seeming to exaggerate due to we can't find similar study.
HPOM was can't estimate when the number of ESs was three in first experiment. This is caused by the random sampling characteristic of RANSAC. The minimum number of an accurate data is presented as a three for the estimation of HPOM in this paper. However, the half of TS that used in HPOM estimation is error. Therefore, the possibility is less than 1% that accurate data will be selected in all three times. In addition, the number of iterations was decided by the number of ESs in RANSAC. The max iterations are 35 times when the number of ESs is three.
Consequently, HPOM estimation is almost impossible by in probability when ES is three.
The processing time was increase whereas the accuracy was decrease when the number of ESs was set excessively. owing to the characteristic of RANSAC that is repeat an estimation process until an optimal model is found.
MDG affects the accuracy of HPOM and the processing time. The processing time is consumed over 60 second when MDG is set to 0. On the other hand, MDG is set to over 0, the processing time is consumed under 1 second.
MDG accurately satisfied the goal that a decreasing the processing time. The accuracy of HPOM is almost constant when MDG is set to 0 regardless of AAE. However, the accuracy of HPOM is irregular when MDG is not 0. It seems to reduce the accuracy of HPOM estimation. This result is thought to have been caused by a lack of discernment in the method using MDG. This method only compares the angle of straight lines without considering the intersection point in ORCP inference section. Thus, a straight line that does not pass through ORCP can be selected as the correct answer, so that the straight line of the same angle passing through the ORCP can be ignored. Therefore, the accuracy of HPOM appears irregular regardless of MDG value when MDG is greater than 0. In addition, it is thought that irregular results were emphasized due to the characteristic of RANSAC that selecting samples randomly. In conclusion, MDG effectively reduces the processing time, but the accuracy of HPOM can be reduced.
The accuracy of the proposed method uses RANSAC method, it depends on the quality and amount of data used for HPOM estimation. In particular, the amount of data may be obtained by repeated photographing, while the quality of data is affected by camera performance. This is because, if the number of pixels constituting the pupil is small in the image that photographed the face, the analysis for HPOM estimation is impossible due to the pupil shape almost unchanged even if the gaze direction changes. In addition, since the proposed method is consisted by linear analyzes the location and shape of the pupil and estimates HPOM, the data of the pupil staring at least three different directions are required. Therefore, there is a limitation in that HPOM cannot be estimated if the data was consisted by without the eye-gaze direction change. Moreover, in this paper, HPOM was limited to a sphere, but later research is planned to be expanded to ellipsoids.

Conclusion
In this paper, we suggested a novel method of HPOM estimation for eye gaze tracking. A pupil pose was analyzed in a 3-dimensional space by using a pupil shape, and PCP. Subsequently, ORCP was inferenced by using an analyzed pupil data. Afterwards, HPOM was estimated through the process of acquiring radius based on ORCP.
The entire process is conducted using RANSAC to acquire a suitable HPOM. A simulation experiment was performed to confirm the performance of the proposed method. The error of HPOM estimation appears under 1 pixel in the first experiment both ORCP and a radius. In addition, it was confirmed that the process of estimating HPOM can be conducted as a real time in the second experiment. However, MDG, a variable suggested in this paper, affected to the processing time and accuracy of HPOM. The error of HPOM was increased coincide with the processing time was reduced when MDG was set to over 0.
The proposed method was able to overcome the common problem of existing studies that could not find the rotational center of an eye-ball. In addition, HPOM may be secured in real time even if the subject moves.
However, the proposed method has limits as applicable only when HPOM is a sphere. In the future, we plan to research considering that the actual HPOM is an ellipsoid.

Theoretical background
The pupil is a circular hole on the surface of the eyeball surrounded by the iris. Through the pupil, information about the shape or color of an object is transmitted to the visual nerve. It is observed as black circle regardless of race, age, or gender. It is well known that when the eye gazes at an object, it rotates about the central point inside the eye so that the pupil faces the object.
Joseph L. Demer and Robert A. Clark have observed the rotational movement of the eye using magnetic resonance imaging. As a result, ORCP reported an average position change of 0.77 when the eye moved in the abduction direction and 0.14 when the eye moved in the adduction direction [15]. Their study can be interpreted that the pupil moves along an ellipsoid orbit and the distance between PCP and the Real ORCP changes according to the gaze direction. However, if one uses a camera to observe eye movement, the pupil will move along a spherical orbit. This is because when an object is photographed with a camera, the image of the object passes through the lens and distortion occurs according to the law of refraction. In addition, the image of the distorted object is projected on the camera's image sensor and converted into a digital image through an analog-to-digital converter [16]. Due to characteristics of the lens and an analog-to-digital converter process, the camera represents an area of several millimeters to several centimeters as 1 pixel depending on the distance from the object, accompanied by distortion of the object's shape. Based on this, the HPOM is expected to converge to a sphere. It can be estimated using the pupil observed as an ellipse. In order to estimate the three-dimensional HPOM, the shape of the pupil, which is deformed into an elliptical shape according to the gaze direction, can be analyzed and converted into three-dimensional coordinates. The positional relationship between the ORCP and threedimensional the PCP can then be analyzed [10].

Eye movement observation environment
As shown in Figure 1, the position of the ORCP can be kept constant. The camera faces the front of the face.
Camera and face movements are fixed. After that, HPOM reconstruction can be performed by securing the projected pupil onto camera (PPC) in the configured environment.

HPOM design
When HPOM is set as a sphere, the relationship between ORCP and PCP satisfies (1) of the sphere. It can be expressed in a spherical coordinate system as shown in Fig. 2. The gaze direction is indicated by the g-axis in the direction from the ORCP to the PCP. When the g-axis is oriented in any direction, the positional relationship between PCP and ORCP is described by radius R and direction angles θ and ф in the spherical coordinate system.
D is the line segment that appears when the line connecting ORCP and PCP is projected on the xy-plane of the spherical coordinate system. It can be calculated as shown in (2) by a trigonometric function. θ means the angle between the g-axis and the z-axis. It is expressed in the 0 to π range. ф means the angle formed by the line segment D with the x-axis. It is expressed in the range of 0 to 2π.

Fig. 3. Relationships between PCP and ORCP in Spherical
Coordinates.

PPC analysis
The pupil is observed as an ellipse according to the gaze direction. It can be assumed to be a circle with zero eccentricity when the camera is viewed from the front [10]. Based on this, the two prerequisites were set to estimate the 3D position of the PCP as follows: The pupil is a disk with zero eccentricity located on the surface of the HPOM and perpendicular to the g-axis.
The xy-coordinate system of the camera and HPOM are parallel with a certain distance along the z-axis.  Based on the established premise, the three-dimensional position of the pupil was estimated by analyzing the PPC observed in different shapes depending on the gaze direction. When the gaze direction is changed while the ф is fixed to 0, the HPOM observed by the camera can be represented by the x-z section shown in Fig. 3. The shape of the pupil projected on the camera is shown in Fig. 4. When the gaze direction is perpendicular to the camera, θ becomes 0. The pupil disk is positioned as shown in Fig. 3A. It is projected as a circle with zero eccentricity as shown in Fig. 4A. On the other hand, when the gaze direction is not perpendicular to the camera plane, the pupil disk is positioned as shown in Fig. 3B due to a three-dimensional rotational movement based on the ORCP according to the premise. At the same time, the distance d between the PCP and the camera increases.
Due to this, the three-dimensionally rotated and moved pupil disk is projected orthogonally to the camera, resulting in an elliptical shape as shown in Fig. 4B. Moreover, as it is accompanied by a change in the distance d between the PCP and the camera, perspective distortion occurs when the pupil is projected onto the camera.
If the gaze direction changes from the position of B in Fig. 3 while θ is fixed, the PPC appears in the form of B rotated along the z-axis relative to the ORCP as shown in Fig. 4C. Therefore, θ is the rotation size angle of the eye (RSA). It causes a change in size due to change in the eccentricity of the PPC and a change in the distance on the z-axis. ф is the rotation direction angle of the eye (RDA). It causes rotation of the PPC. In addition, it can be seen that the short axis of the elliptical-changed PPC is formed at the same angle as the RDA.

3D rotation of the pupil disc
Previously, the principle that the eccentricity and rotation of PPC change according to RSA and RDA using HPOM was analyzed. Based on analysis results, a three-dimensional rotation model of a disk (RMD) was constructed to analyze the pose in the three-dimensional space of the pupil. RMD is a model for observing a threedimensional pose with a camera that changes due to the rotation of each axis of the disk in an environment where the xy-coordinate system of the camera and the disk are synchronized and a constant distance is formed along the z-axis. When the disk is perpendicular to the z-axis of the camera, the initial pose of the RMD is set and the pose in which the disk is rotated 3D with respect to the center point and the shape projected on the camera are analyzed. Fig. 5A and 5B are obtained by changing the y-axis rotation angle in the initial pose of the RMD. In Fig. 5A and 5B, the short axis length appears in inverse proportion to the rotation angle of the y-axis due to the relationship shown in Fig. 6. It shows the disk pose before and after the y-axis rotation on the xz-plane, showing the formation principle of the diameter and of the disk projected on the camera. Since the disk is parallel to the camera plane in the initial pose, the angle between the rotationally changed disk and the camera plane is the same as the rotation angle of the disk. Therefore, the length of the diameter projected on the camera plane when the rotational change occurs as much as θ can be calculated using (3) based on the trigonometric function.

Results of
In addition, the diameter of the disk tangent to the y-axis where the rotation occurred is the same as the diameter of the disk regardless of the size of the rotation angle. It appears as a long radius in the ellipse. Therefore, if it is possible to secure , the short axis of the projected ellipse after a three-dimensional rotation change, and , the long axis, (3) can be summarized with respect to the rotation angle of the disk, θ, and calculated with (4).
The length of the long axis was constant at even though θ, the rotation angle of the y-axis, was different.
This was because there was no change in the distance between the camera and the z-axis in the part of the disk that was in contact with the y-axis where the rotational change occurred. Based on this, it was expected that the length of the long axis would always be constant even if the rotation of the other axis occurred. To confirm this, the result projected on the camera by rotating each axis at any angle is shown in Fig. 5C. While lengths of the short axis and long axis of the ellipse and the angle formed with the x axis changed, the length of the long axis was always the same as .
By reflecting the RMD analysis to the PPC change according to the pose change of the pupil disk analyzed previously, it can be interpreted as shown in Fig. 7 according to the premise. In Fig. 7, points O, P, and Q indicate ORCP, PCP, and the position when a line is drawn perpendicular to the z-axis in PCP, respectively. Point R is the position where the z-axis meets when a perpendicular is drawn parallel to the rotationally changed pupil disk in PCP. Since the sum of interior angles of a right triangle is always 180 degrees, angle of QRP can be calculated using (4). Due to the same principle, it can be seen that the rotation angle θ of the pupil disk is the same as RSA.
Therefore, if lengths of the short axis and the long axis of the PPC are known, RSA can be calculated according to (5). Thus, the three-dimensional position of the PPC can be approximated by the center point of the PCP and the RSA Fig. 6. The projected result as an ellipse onto the camera after a disk has rotated each axis.  It can be seen that the point of intersection of all circles is the same. Based on this, in order to secure a straight line passing through the intersection point of circles, (9) of the circle centered on each point P is combined and arranged as shown in (10). Through the process of (10), the radius of the unknown HPOM is canceled. It does not affect the position inference of ORCP. However, the calculation result of (10) is arranged as a straight line equation as shown in (11) when the difference of constant K in (10), the coefficient of the quadratic term, is 0.
However, in the form of a new circle equation as shown in (12), when it is not 0, it is sorted out. Since at least one of the intersections of two simultaneous circles is always an ORCP, the equation of a newly secured circle also forms an intersection in the ORCP. Therefore, equations of all circles newly obtained through the simultaneous process are divided and normalized by k in (11), which is the coefficient of each quadratic term. When equations of the normalized circle are combined, the quadratic term is canceled and all are arranged in (11) of a straight line.
Finally, the ORCP estimation is completed through the process of finding intersections of all obtained straight lines. In order to find the intersection of all straight lines, it is divided into a coefficient term, an unknown term, and a constant term. It arranges as shown in (13) as a determinant. Since the unknown term is the position on the xy-plane of the ORCP to be estimated, the ORCP is finally secured by solving (14) organized for the unknown term by finding the inverse matrix for the constant term using the Singular Value Decomposition method [17].
Ax + By = C (11)  The position of the ORCP on the xy-plane was secured using the previously approximated position of the PCP.
All PCPs centered on the secured ORCP satisfied (2) in the xy-plane. The relationship is shown in Fig. 9. The distance between ORCP and PCP on the xy-plane showed the same change as the sine wave according to the change in RSA. The maximum distance was equal to the radius of HPOM when RSA was 90 degrees. Based on this, the approximate three-dimensional position of the PCP can be reflected in (2) and summarized as (15). The radius of the HPOM can be finally obtained by solving (16), which is obtained the inverse matrix of the coefficient term by using the Singular Value Decomposition method [17]. Robert C. Bolles [18]. The RANSAC method has a feature of repeatedly performing mathematical model estimation by randomly selecting the model estimation sample (ES) from the total sample (TS) and deriving an optimal result. As a result, even if the TS contains error samples, it shows high accuracy. However, there is a disadvantage in that the computational load is large in proportion to the number of TSs and ESs. Therefore, it is necessary to set the appropriate number of TS and ES to minimize the computational load with a good performance to derive an optimal model. HPOM fitting repeats the Hypothesis and Verification steps according to the sequence shown in Fig. 10. In the hypothesis stage, ES is randomly selected from TS and HPOM is estimated according to the proposed theory. The number of straight lines that can be secured in the ORCP estimation process is proportional to the number of ESs.
Therefore, when ORCP is estimated using many samples, the equation of a straight line with the same slope can be obtained. Lines with the same slope may cause computational load and errors because innumerable intersections with each other appear. Therefore, it is necessary to consider the slope in the process of securing the equation of the straight line. Minimum Difference of Gradient (MDG) is reflected as a variable to exclude equations of straight lines with the same slope. When the difference between the slope of the newly secured line and the slope of the previously secured line exceeds the MDG, the equation of the line is used for ORCP estimation.
If MDG is set to ±1 degree, the maximum number of straight lines obtained is 359 even if there are countless samples.
In the verification step, error determination is performed. TS samples satisfying the HPOM estimated in the hypothesis step are classified as inliers. The quantity is secured as a cost. Error determination is performed by reflecting the ORCP, the distance on the xy-plane of the determination object sample, and the RSA of the determination object sample in (2). However, since the shape of the pupil observed using the camera might be distorted depending on characteristics of the lens, an error of RSA may occur. Therefore, the allowable angle error (AAE) was reflected as a range variable to determine the allowable error range of the distance on two-dimensional coordinates of the sample to be judged. After that, the measured error of the judgment target data that could be secured according to (2) with the distance on the xy-plane between the ORCP and the judgment target data was compared and classified as an inlier if included in the AAE and an outlier if it was not included in the AAE. If the cost obtained by performing error determination is large compared to the maximum cost, samples classified as inliers are stored. This is because the larger the cost obtained through error determination, the more similar the estimated HPOM to the actual HPOM. This process is repeated and the optimal HPOM is re-estimated using the inlier obtained when the cost is at maximum. The fitting process is completed. Fig. 11. The process of RANSAC to estimate HPOM.