Performance Evaluation of Direction Cosine Matrix and Madgwick’s as 3D Orientation Estimation Algorithm for Estimating Ground Contour

—Contour interval is elevation difference between two contour lines that are adjacent and parallel. Ground Contour interval is an important information that must be provided in a construction. Ground contour interval estimation problem could be viewed as a 3D trajectory estimation problem. One framework, called dead reckoning, is often used for motion (position and orientation) tracking of a mobile systems over time. Dead reckoning requires a precise 3D orientation estimation algorithm. Two competing algorithms are Direction Cosine Matrix and Madgwick’s. To ensure the 3D trajectory estimation gives an accurate result, these two algorithms must be carefully and thoroughly evaluated. This research is part of a complete research on the development of a ground contour estimation and only focuses on comparison between two 3D orientation estimation algorithm.


A. Background
Measurement of ground contour maps is a common activity in geomatics concentration. One of the information that can be derived from ground contour maps is the contour interval, which is an elevation difference between two adjacent and parallel contour lines [1,8]. Contour interval is an important information that must be provided in the construction development [2,9]. Without it, building construction design planning becomes inaccurate and might results an unstable, not sturdy building. Unfortunately, contour intervals can't provide any information about the height difference between within two contour lines which involved. This is due to the sparsity of the contour lines in the map. The resolution provided by the contour map also varies depending on its usage.
Ground interval contour estimation problem (henceforth, will be refered as ground contour estimation), can be viewed from another perspective, namely threedimensional (3D) trajectory estimation of a system. 3D trajectory estimation is one branch of computer science that is able to produce a collection of 3D point coordinates in space that are able to represent trace or movement of the system while the system is working [3,10].
One topic that intersects with this problem is called Dead Reckoning (DR). DR is a movement tracking technique of a mobile systems, it tracks the relative position and orientation from time to time. DR is also able to describe the full 3D orientation of the system [4,11], however, over time, the error given by the sensor in each measurement moment will accumulate and would lead to exaggerated error [5]. Measurement errors that is suffered by DR can actually be minimized by implementing a more accureate orientation estimation algorithm. There are two algorithms that are commonly used in the world of robotics, namely Direct Cosine Matrix (DCM) [6,4,12], and Madgwick's algorithm [7,13].
This research is part of a complete research on the development of a ground contour estimation. This study focuses on the implementation and an in-depth performance evaluation of the two 3D orientation estimation algoritms, namely DCM and Madgwick's.

B. Problem Reformulation
Based on the background, some problems can be formulated into several points as follows: 1) How to estimate the 3D orientation of the system? 2) How can the algorithm testing be performed? 3) How can the algorithm be evaluated?

C. Objectives
Based on the formulation of the problem, some of the objectives of this study can be formulated into several points as follows: 1) Implement two 3D orientation estimation algorithms, Direct Cosine Matrix and Madwicks. 2) Using the Inertial Measurement Sensor (IMU) dataset which is available to test wheter the algorithm has been implemented properly. 3) Determine a proper metrics for evaluating numerically (quantitatively) to the two algorithms involved.

D. Contribution
This paper will perform a thorough performance evaluation between orientation estimation algorithms, DCM and Madgwick's, and summarizes the results which would be helpful for future research that involves this algorithm.

A. Orientation Representation
Orientation of a system can be described by several representations, including matrix representation and quaterions representation. Each has its own unique property requirement and operating rules.

1) Rotation Matrix
In linear algebra, rotation matrix is a matrix that is used to perform a rotation in Euclidean space. Rotation matrix that is used to represent an orientation in 3D space is a 3x3 matrix. Rotation matrix has a special properties, which are orthogonal and normal.

2) Quaternions
Quaternion is a four-dimensional (4D) number system that extends the definition of complex two-dimensional (2D) numbers. Similar to the rotation matrix, quaternions are used to perform a rotation in Euclidean space, but it has different set of rules of operations and properties. Quaternions should be represented by a 4D vector with a length of 1. Where

B. Direction Cosine Matrix
A summary of the DCM algorithm is described in Algorithm 1 below. Details of DCM orientation estimation algorithm can be found in its original publication [6].

Input:
Orientation at in rotation matrix Current gravitation acceleration , Current angular velocity , Current magnetic field

Time interval
Output:

Current orientation in rotation matrix
Process:

III. COMPARISON FRAMEWORK
To observe the performance of both Algorithm 1 and Algorithm 2, a test using real data must be performed. Secondary data, derived from [7], are used as input for both of the algorithms.
The available dataset is a pair of files with CSV (comma separated values) extension. The first file, dataset.csv, is a file that contains raw data from 9 Degree of Freedom (DoF) IMU measurements. The dataset.csv file has 6960 data rows including headers. The second file, groundtruth.csv, is a file that contains groundtruth in euler space. The number of rows of groundtruth.csv file has the same number as in the dataset.csv file. TABLE 1 is a description of IMU measurement scenario within the dataset. The scenario is divided into seven steps / stages, arranged in such a way that all IMU's basic orientation are read. The IMU sensor is first placed statically 2 The IMU is moved 90° to the X axis, stops, -90° to the X axis, stops. 3 The IMU was returned to its original position. 4 The IMU is moved 90° to the Y axis, stopped, -90° to the Y axis, stopped. 5 The IMU was returned to its original position. 6 The IMU is moved 90° to the Z axis, stopped, -90° to the Z axis, stopped. 7 The IMU was returned to its original position.
FIGURE 1 is a visualization graph from gyroscope, accelerometer, and magnetometer, sequentially from top to bottom, which derived from the dataset. Red, green, blue lines represent measurements on the X, Y, and Z axis respectively. Performance evaluation and comparison of the two algorithms is done using MATLAB 2016b software on a computer with Ubuntu Operating System 14.04 LTS with spesifications of i5-6200U @ 2.3GHz CPU and 4 GB RAM.

A. Qualitative Evaluation
Qualitative evaluation is performed by representing the collection of estimation data and groundtruth into the same graph.
An algorithm that is said to have low performance is when there is a significant difference between the groundtruth orientation line and the estimated orientation line, indicating the algorithm is unable to estimate the orientation according the actual situation.

B. Quantitative Evaluation
To conduct a quantitative evaluation, orientation estimation error rate calculation to the groundtruth at each given moment must be performed. Errors can be calculated using the error function defined in equation 1 below., where is an l2-norm, is an identity quaternion which has value of 1 for the real component ( ) and 0 for all imaginary component ( , , ), is a ground truth orientation in quaternion, symbol is quaternion product, and is an orientation estimation in quaternion with superscript represent the inverse of quaternion that is defined as, where is a quaternion conjugate from and is an l1-norm from .
in the equation 1 will produce an identity quaternion when the estimation ( ) precisely predict the true orientation ( ) without any mistakes. This condition is the most optimal condition, so that the difference between with the identity quaternion ( ) will gives zero. The error function defined by equation 1, shows that the performance of the corresponding orientation estimation algorithm, i.e. the smaller the value that is calculated using equation 1, the more accurate the estimation made by the algorithm; and vice versa. Value range produced by the error function is zero (0) to one (1). The IMU measurement dataset (see FIGURE 1) at its nine degree of freedom is used as the input for Algorithm 1 (DCM) and Algorithm 2 (Madgwick's). Both of these algorithms produce outputs in the form of collection of orientation estimation in euler angle. These data collection is represented in FIGURE 2 and FIGURE 3
The red, green, and blue lines (see FIGURE 2 and FIGURE 3) represent the orientation estimation for the X, Y, and Z axis, respectively. It can bee seen in the graph, there are three main waves that are formed, each of which is the representation of stage number 2, 4, and 6 that is conducted (see TABLE 1).

B. Qualitative Evaluation
Performance evaluation of Algorithm 1 and Algorithm 2 can be seen by comparing the actual orientation (groundtruth) with the estimated orientation in one graph (see FIGURE 4a and FIGURE 4b). Qualitatively, the estimation (red line) is almost always coincides with the groundtruth (blue line), especially in the three main waves mentioned earlier. This indicates that Algorithm 1 and Algorithm 2 are able to estimate the orientation of the system properly.
Exciting things happening in sample ±3000 to ±4000, i.e. there is a sudden spike that only happens in the estimated roll (FIGURE 4 -first column) and estimated yaw (FIGURE 4 -third column). This spike was not found in the estimated pitch (FIGURE 4 -second column).
This spike is an artifact of a gimbal-lock, a phenomenon that only occurs in euler angle representation, which is a specific situation where euler angle is not able to represent the full 3D orientation. The specific situation mentioned is when the system orientates itself towards 90° on the Y axis (pitch). This spike does not mean that Algorithm 1 and Algorithm 2 fail to estimate orientation, it just means that euler angle representation is not able to decribe a situation when the system orientates it self 90° on the Y axis (pitch).
Euler angle is indeed the most intuitive orientation representation of a system to be analyzed by humans, but due to limitation possesed by this representation, the performance of Algorithm 1 and Algorithm 2 should be done using another orientation representation. FIGURE 5 and FIGURE 6 are orientation estimation and groundtruth comparison in quaternion representation. Red, green, blue, magenta lines are the w, x, y, z components of quaternions, respectively.
The results of qualitative evaluation (graphical) do not show significant differences between the two algorithms. Altough if we look carefully, there is actually a slight deviation performance, but it is quite difficult to see by human eye. Therefore, quantitative testing had to be conducted.

FIGURE 5 DCM Orientation Estimation Comparison with
Groundtruth using Quaternion Representation.

FIGURE 6 Madgwick's Orientation Estimation Comparison with
Groundtruth using Quaternion Representation.

C. Quantitative Evaluation
The quantitative analysis was conducted in two related algorithms. To strengthen the analysis results, the two algorithms are compared at different noise levels. Noise reference (reference / baseline noise) became the basis for the comparison. Baseline noise is estimated by measuring the standard deviation ( ) of the mean-normalized data on each axis (x, y, z) on each sensor (accelerometer, gyroscope, magnetometer). FIGURE 7 shows the original measurements (red lines) and measurements with noise (green lines) at two noise levels, and .

FIGURE 7 Comparison between Original and Noise Added
Measurement with 2σ (left) and 5σ (right) FIGURE 8 shows the comparison of error values owned by the two algorithms using dataset inputs that are subject to 11 different noise levels, from to . Estimation results in each noise level are compared with groundtruth using the error function (equation 1). To ensure quantitative measurements, the two algorithms were run n times ( in this study) so as to produce a collection of error values. The mean of the error is plotted into a graph and becomes the reference for this quantitative analysis.

FIGURE 8 DCM and Madwick's Comparison in Resistance to
Noise.
It is shown by FIGURE 8, the DCM algorithm (blue line) tends to always have a lower error value than the Madgwick algorithm (red line). Therefore, it can be concluded that the DCM algorithm has the advantage in resistance to noise. FIGURE 9 shows the comparison of the duration required for each algorithm in completing its operations for all data available in the dataset. Duration is measured in seconds. The blue bar indicates the duration of pure operation required by each algorithm, i.e. without any data storage into variables or other things. The green bar indicates the operation duration with a storing process of the estimation orientation results into a variable. The yellow bar shows the operation duration with a transforming process from the body-frame coordinate system to the world-frame coordinate system (a process that must be done if we want to represent the orientation in the observer coordinate system). As can be seen from FIGURE 9, the Madgwick algorithm (left) is superior in speed compared to the DCM algorithm (right). This conclusion is consistent with one of the claims in the original publication of Madgwick's Algorithm [6]. FIGURE 10 shows the comparison of the amount of memory needed for each algorithm to store the entire collection of estimated system orientation values from the dataset. Memory size is measured in bytes. As can be seen from FIGURE 10, the Madgwick algorithm (left) is superior in memory efficiency compared to the DCM algorithm (right). This conclusion also agrees with one of the claims in the original publication. This is because Madgwick's works in quaternion space, which is a four dimensional vector (requires 4 floating point values) while DCM works using a transformation matrix (requires 9 floating point values).
Generally, in this study, the two orientation estimation algorithms are compared using three dimension of evaluations, namely error rate (and their resistance to noise), the execution duration, and the amount of memory needed. These three comparison is summarized in the TABLE 2 below, with sign indicates superiority and sign indicates inferiority. To perform ground contour estimation, one part of the algorithm that must be evaluated carefully is the orientation estimation algorithm. This algorithm is the key to estimating 3D coordinates / system trajectories, in other words, ground contours. Two algorithms that are evaluated in detail in this study are (1) the Direct Cosine Matrix (DCM) Algorithm, and (2) Madgwick Algorithm.
The test was conducted using the IMU sensor measurement dataset and the evaluation was performed with two different approach, qualitative and quantitative. Qualitatively, both algorithms are able to estimate orientation based on predetermined IMU measurement data. The difference is not significant, so quantitative testing is carried out.
Based on three quantitative tests, the DCM algorithm is superior in accuracy and resistance to noise, while Madgwick's algorithm is superior in execution duration and memory efficiency. Each algorithm does have advantages and disadvantages, but the conclusions from this test can be drawn as follows; 1. If the available hardware has a small and limited specification (example: a microcontroller), Madgwick's algorithm becomes a preference, 2. In contrast, if the available hardware has high specifications and is flexible (for example: a computer), the DCM algorithm becomes a preference.
Because for future research (estimation of land contours), a computer will be used as the machine for applying the algorithm, DCM algorithm was chosen to be the orientation estimation algorithm. The main priority in this research is to produce contour plots with a high degree of accuracy. The duration and amount of memory are the second priority because computer specifications can be increased according to algorithm requirements.

DECLARATIONS
* Ethics approval and consent to participate Manuscript reporting studies is not involving human participants, human data or human tissue "Not applicable" in this section" * Consent for publication Our research does not involve human participation; our study only compares a number of algorithms to develop more optimization algorithms "Not applicable" in this section" * Availability of data and material All data is taken from the experiment itself using the equipment that we made. The data used for the experiment is not harmful to the public. Ground contour data for simulation data is used to test algorithm optimization "Not applicable in this section" * Competing interests Currently it is not receiving research funding from the Indonesian Government due to the covid-19 outbreak. The publication fee will be carried out independently by authors, , so we ask for a relief in financing. and our request was approved by Springer without paying publication fees.