Simultaneous measurement of multipoint mechanical vibration for thin-walled parts using monocular machine vision optical flow tracking

: Thin-walled parts (TWPs) have been widely employed in various industry fields and put forward a specific requirement of simultaneous multipoint vibration measurement for their dynamic behaviours understanding. The development of machine vision theory and technology enable the machine vision-based technique to measure the vibration of mechanical equipment. Lucas-Kanade based template tracking (LKBTT) can provide a robust, longer term motion estimation of the interesting object, but lacks spatial resolution and also deeper use of contextual cues. Lucas-Kanade based optical ﬂow tracking (LKBOPT) is usually used for frame-by-frame feature-level motion estimation, and has the potential to satisfy the vibration measurement requirements of TWPs. LKBOPT has not been introduced to vibration measurement yet, and also had the challenge of long-term motion estimation. In order to satisfy the simultaneous multipoint vibration measurement requirements of TWPs, a coarse-to-fine tracking method is developed by introducing LKBOPT to vibration measurement and combining it with LKBTT. A region of interest detection algorithm is investigated to acquire the tracking template according to the measurable object. LKBTT is introduced to realize the coarse tracking and obtain the vibration information of the tracking template. An improved LKBOPT is investigated to realize the fine tracking and acquire the vibration information of multipoint inside the tracking template simultaneously. Finally, the simultaneous measurement system of multipoint mechanical vibration for TWPs is utilized with a high-speed camera system based on both Python and open source computer vision (OpenCV) libraries. The vibration measurement performance of the proposed method has been experimentally verified with two experiments under laboratory conditions by comparing the results with those using accelerometers. The experiment has shown that the proposed method is effective in the remote simultaneous multipoint vibration measurement of TWPs with satisfactory accuracy and efficiency.


Introduction
Thin-walled parts (TWPs) have been widely employed in various industry fields, especially in aerospace, automobile and energy industry owing to their lightweight and high specific strength properties [1][2] . However, TWPs are apt to deform and vibrate excessively during the running because of the low rigidity. The over deformation and vibration will result in fatigue damage and even breakdown after long-time running. Therefore, it is very necessary to understand the dynamic behaviours of TWPs for dynamic optimization, structural damage identification, etc. Furthermore, TWPs are flexible bodies and the vibration characteristic is different from that of rigid bodies which vibrate all in the same way. So, the simultaneous multipoint vibration measurement is the specific requirement of TWPs as to the rigid body.
Mechanical vibration measurement is essential to understand the dynamic behaviours of a mechanical equipment [3] , it also serves as the basis for various engineering practices such as natural frequency or it can only provide sparse and discrete measurements. Secondly, the size of the template is a little big which will inevitably result in low resolution spatial sensing. Finally, only the motion information of the entire template can be obtained by this approach, the motion information of the interesting object or specific multipoint inside the template cannot be extracted. In conclusion, LKBTT cannot satisfy the vibration measurement requirements of TWPs.
Lucas-Kanade based optical flow tracking (LKBOPT) is usually applied for frame-by-frame featurelevel motion estimation, which means that LKBOPT can take feature points as tracking object. It has the potential to satisfy the vibration measurement requirements of TWPs, nevertheless it has not been introduced to vibration measurement yet. Furthermore, LKBOPT has been widely used as a motion feature tracking, long-term motion estimation is still a challenge for optical flow estimation even when modern optical flow approaches are employed [27] , because it is based on brightness constancy assumption.
In this case, introducing LKBOPT to vibration measurement and combining it with LKBTT would be useful to satisfy the simultaneous multipoint vibration measurement requirements of TWPs. As a result, a coarse-to-fine tracking method is developed, and a simultaneous measurement of multipoint mechanical vibration for TWPs using monocular machine vision optical flow tracking is proposed. An interesting object detection algorithm is investigated to acquire the image region of the measured object from an initial frame of the video image sequence and then the image region is taken as the tracking template. LKBTT is introduced to achieve the coarse tracking and obtain the motion information of the tracking template. An improved LKBOPT is investigated to realize the fine tracking and provide the motion information of the multipoint inside the tracking template region. Finally, the simultaneous measurement system of multipoint mechanical vibration for TWPs is utilized with a high-speed camera system based on both Python and open source computer vision (OpenCV) libraries. The vibration measurement performance of the proposed method has been experimentally verified with two experiments under laboratory conditions by comparing the results with those using accelerometers. The experiment has shown that the proposed method is effective in the remote simultaneous multipoint vibration measurement of TWPs with satisfactory accuracy and efficiency.
This paper is organized as follows: the theory and method of the proposed simultaneous multipoint vibration measurement system are presented in Section 2. Two experiments for performance verification are presented in Section 3, and conclusions are drawn in Section 4.

Theory and method
The theory and method of the proposed simultaneous multipoint vibration measurement system are introduced in this section.
The flowchart of the proposed simultaneous multipoint vibration measurement method is shown in Fig. 1. Compared with LKBTT vibration measurement approach, the proposed simultaneous multipoint vibration measurement method has some advantages. Firstly, the proposed method can realize simultaneous multipoint vibration measurements for TWPs. Secondly, the size of the template is decided by the detected object of interest which provides contextual cues for LKBOPT. Finally, the motion information of the object (or specific multipoint inside the template) can be extracted.
Proposed coarse-to-fine tracking method

Coarse-to-fine tracking principle
The basic procedure of the coarse-to-fine tracking method is shown in Fig. 2, where Fig. 2 (a) depicts the initial frame of the video image sequence. The object of interest is detected and then taken as the tracking template. The features inside the template are detected in accordance with the requirement of improved LKBOPT. Fig. 2  . All of the layers i S are related to meaningful regions which usually denote some level of semantic and can be considered as a sub-set of an image. Supposing each one of these layers i S has an assigned motion vector i w at location p with respect to its parent in the hierarchy, the total motion of the pixel p can be expressed as follows: where p ε is the motion vector of the region or the whole object containing the point p .
In this case, the motion computation of each layer in the hierarchy is supported by the calculation of the predecessor layer. For most cases, a simple scheme with two layers will be sufficient to obtain accurate tracking result. Thus, simplifying the Eq. (1), we can write: where O w is the differential motion of the point p with respect to the motion of the entities the point belongs to, p ε will be obtained with LKBTT and O w will be acquired by improved LKBOPT.

Template detection algorithm
The task of the template detection algorithm is to detect the measured object within initial fame of the video image sequence and to draw the bounding rectangle with minimum area around it. The region contained inside the bounding rectangle of the image is then taken as the tracking template. The flowchart of the template detection algorithm is shown in Fig. 3. The total algorithm was coded using OpenCV functions and the Python. The details can refer to the OpenCV-Python Tutorials.  x as closely as possible [17] . In LKBTT algorithm, the warp function plays a very important role in tracking different types of motion. For instance, 2-Dimension affine transformation is widely used in general tracking. It consists of six independent parameters and takes many linear transformations into account, including translation, rotation, shear mapping, and scaling. In order to improve the algorithm performance, the warp function in Reference [22], and only comprises two independent parameters since the authors believe that the movements of structures are mostly in-plane motions with a limited range, and deformation and rotation can be neglected in a short capture. This simplification is sometimes appropriate for application to large-scale structures, but it has limitations for measurement of TWPs, so the un-simplified warp function ( ; ) W x p has been used in the proposed simultaneous multipoint vibration measurement method.

Improved LKBOPT algorithm
If I and J are two sequential grayscale images, x , then because of the problem of aperture, it is necessary to define the notion of similarity in a neighborhood sense. If x  and y  are two integers, the residual function () ε d can be defined as follows: where the similarity function is measured on an image neighborhood of size ( (3). A tradeoff exists between local accuracy and robustness when choosing the integration window size.
The initial guess of the flow vector L g is used to translate the image patch in the second image J .
The optical flow solution is then available after the finest resolution optical flow computation: This solution may be expressed in the following extended form: The clear advantage of a pyramidal implementation is that each residual optical flow vector L d can be kept very small whilst computing a large overall pixel displacement vector d . Assuming that each elementary optical flow computation step can handle pixel motions up to dmax, then the overall pixel motion that the pyramidal implementation can handle becomes At every level L in the pyramid, the goal is finding the vector L d that minimizes the matching function L ε defined as follows.
Since the same type of operation is performed for all levels L , the superscript L is dropped and the new images A and B are defined as follows [28] : The domains of definition of ( , ) A x y and ( , ) At the optimum, the first derivative of ε with respect to v is zero: After expansion of the derivative: The quantity ( , ) In practice, the Sharr operator is used for computing image derivatives [28] . Eq. (12) can then be written:  (19) Then Eq. (18) can be rewritten: Therefore, according to Eq. (18), the optimum optical flow vector is as follows: This expression is valid only if the matrix G is invertible; that is equivalent to saying that the image ( , ) A x y contains gradient information in both x and y directions in the neighborhood of the point p , and this is the requirement that the tracking features need to satisfy. This is the standard Lucas-Kanade optical flow equation, which is valid only if the pixel displacement is small. In practice, to get an accurate solution, it is necessary to iterate multiple times on this scheme.
Supposing k is the iterative index, initialized to 1 at the very first iteration. The algorithm is described recursively as follows: at a generic iteration The spatial derivatives x I and y I (at all points in the neighborhood of p ) are computed only once at the beginning of the iterations following Eq. (15) and (16). Therefore the 2  2 matrix G remains constant throughout the iteration loop. This constitutes a clear computational advantage.
The only quantity that needs to be recomputed at each step k is the vector k b that captures the amount of residual difference between the image patches after translation by the vector The iterative scheme goes on until the computed pixel residual k  is smaller than a pre-set threshold or a maximum number of iterations is reached. On average, 5 iterations are enough to reach convergence [28] . At the first iteration ( 1 k  ) the initial guess is initialized to zero: Assuming that K iterations were necessary to reach convergence, the final solution for the optical flow vector L v  d is: This vector minimizes the error function described in Eq. (9). This ends the description of the iterative Lucas-Kanade optical flow computation. The vector L d is fed into Eq. (5) and this overall procedure is repeated at all subsequent levels 1, 2, ,0 LL  L . The improvements made to LKBOPT can be summarised as follows. Firstly, the complicated pyramidal implementation is not necessary in the proposed approach. The complicated pyramidal implementation of LKBOPT is needed to solve the large displacement problem, but because the large displacement is measured with LKBTT algorithm, the displacement between template 0 T in initial image 0 I and tracked template i T in current image i I is very small. Secondly, the optical flow estimation is calculated between template 0 T and template i T , rather than between the previous image I and current image J . The improvements enhance the robustness of the proposed method and solves the problem of long-term motion estimation.

Experiment and results
In this section, the experiment rig is built and two experiments are performed under laboratory conditions to verify the performance of the proposed simultaneous multipoint vibration measurement system.

The proposed simultaneous multipoint vibration measurement system
As shown in Fig. 4(a), the developed simultaneous multipoint vibration measurement system consists of a notebook computer connected to a video camera with a telescopic lens. As shown in Fig. 4(b), the telescopic lense has a large optical zoom capability and can be adjusted appropriately to capture mechanical equipment vibration at different distances. The video camera uses a CMOS sensor as the image receiver and can capture 8-bit grayscale images at high speed, which are streamed into the notebook computer through a USB 3.0 interface. The camera frame rate can reach up to 600 fps when the image resolution set to 640×480 pixels. The accuracy and capacity of the proposed system is also affected by the performance of computer, so the specifications of the laptop computer are provided. Technical specifications of the system are shown in Table 1. The proposed algorithms have been integrated into the developed simultaneous multipoint vibration measurement system based on both Python and OpenCV libraries. The programmed software consists of calibration module and video analysis module. The calibration module is used to calculate the actual size that single pixel occupies on the target, and the parameters pixel size and focal length will be used. Any video image sequence captured by the high-speed video camera can be transferred to the video analysis module, and the motion information is then extracted by the proposed algorithms.

Vibration measurement accuracy and capacity verification experiment
The vibration measurement accuracy and capacity of the proposed system was first evaluated through a shaker vibration measurement experiment. In this experiment the proposed system was experimentally evaluated against a traditional accelerometer sensor. The vibration was recorded simultaneously by the accelerometer and the high-speed camera system to permit direct comparison. The experimental rig is shown in Fig. 5(a). An accelerometer (Model CA-YD-185 by SINOCERA Piezotronics Inc.) was fixed to a modal shaker (Model JZK-5T by SINOCERA Piezotronics Inc.). The shaker was driven by a signal generator (Model YE1311ET by SINOCERA Piezotronics Inc.) and the frequency and amplitude of the driving signal were set manually. The accelerometer vibration signal was sampled with a dynamic measuring (Model YE6231 by SINOCERA Piezotronics Inc.) and streamed into a computer through USB cable where it was recorded by general control & analysis software (Model YE7600 by SINOCERA Piezotronics Inc.). In parallel, the vibration of the accelerometer sensor was also recorded with the highspeed camera. The video images captured by the camera were digitized into 640×480 pixel images in 8-bit grey scales and also streamed into the computer through USB interface. The entire experimental setup is shown in Fig. 5(b). The sampling frequency of the general control & analysis software was 375Hz, and the frame rate of the camera was set to 375 fps. The sampling time for both devices was 16 seconds. The motion information of the accelerometer sensor was extracted using the proposed system on the attached computer. An image captured by the camera is displayed in Fig. 5(c), in which the red box is selected as the template and the green points are selected as the tracking features. To evaluate the measurement capability from low frequency to high frequency, the vibration frequency of the shaker was set at successive values of 1Hz, 4Hz, 7Hz, 10Hz, 30Hz, 50Hz, 70Hz, 90Hz, 110Hz, 130Hz and 150Hz all at an amplitude of approximately 6mm. Since the frequency is set by an adjusting knob, the frequency cannot be fixed precisely at a certain value, the exact frequency being delivered was obtained by processing the vibration signal from the accelerometer. The displacement signal was acquired by the proposed system, whilst acceleration signal was obtained by the accelerometer. The acceleration signal was converted into its corresponding displacement for comparison. After conversion, both of the displacement signals were processed by FFT, and the amplitude frequency spectrums were plotted, as shown in Fig. 6 (a) to (k). The frequency error between the two methods is shown in Fig. 7. The results agree well, with a maximum frequency error of 0.06Hz. The displacement time history measured by the proposed system is compared with that measured by the accelerometer, and this was shown in Fig. 8. The results correlate well, with an average amplitude error of 0.012 mm. It is hence concluded that, the proposed system can accurately measure such vibration within the frequency range 1 Hz to 150Hz.  This vibration measurement accuracy and capacity verification experiment demonstrates that the proposed system can measure dynamic vibration with good accuracy. Nevertheless, because the improved LKBOPT is a sub-pixel optical flow estimation method, its accuracy is related to the pixel resolution but not limited by it.

Simultaneous multipoint vibration measurement experiment of a typical TWPs
The vibration measurement effectiveness and capacity for TWPs of the proposed system is then evaluated through a simultaneous multipoint vibration measurement experiment of an aircraft T tailplane model. An aircraft T tailplane model is built with stainless steel plate of 1mm thickness. The proposed system is experimentally evaluated against the traditional accelerometer sensor in this experiment too. The vibration can be recorded simultaneously by the accelerometer sensor and the high-speed camera system for comparison. The shaker with aircraft T tailplane is shown in Fig. 9(a). Two accelerometer sensors are mounted on both sides of the T plane model, and the whole model is fixed on the shaker. The shaker is driven by the signal generator. Then the accelerometer vibration signal is sampled at 375Hz with the dynamic data acquisition system and streamed into the computer through the USB cable and recorded by the general control & analysis software supplied with the system. Meanwhile, the vibration of the whole model is also recorded with the high-speed camera. The video images captured by the camera are digitized into 640×480 pixel images in 8-bit grey scales at 375 fps and streamed into the computer through an USB 3.0 cable. The whole experimental rig is shown in Fig. 9(b). The sampling time of both two are set as 16 second. The motion information of the accelerometer sensor is extracted by performing the proposed system on the connected computer. A typical image captured by the camera is displayed in Fig. 9(c), in which the red box is selected as the template and the green points are selected as the tracking features. The vibration frequency of the shaker is set as about 30.9Hz. And then the acceleration signals are transferred into displacement signal for comparison too. Then both of the displacement signals are processed with FFT. The displacement time history measured by the proposed system are compared with those measured by the accelerometer sensors, which is shown in Fig. 10(a) and (b), the amplitude frequency spectrum is compared and shown in Fig. 10(c) and (d). The results agree well, and the frequency error is 0.04 Hz and the average displacement error is 0.025 mm. Therefore, the proposed system could accurately measure multipoint vibration of TWPs simultaneously, having demonstrated its good measurement capacity.

Conclusion
In this study, in order to satisfy the simultaneous multipoint vibration measurement requirements of TWPs, a coarse-to-fine tracking method is developed by introducing LKBOPT to vibration measurement and combining it with LKBTT. A region of interest detection algorithm is investigated to acquire the tracking template according to the measurable object. LKBTT is introduced to realize the coarse tracking and obtain the vibration information of the tracking template. An improved LKBOPT is investigated to realize the fine tracking and acquire the vibration information of multipoint inside the tracking template simultaneously. Finally, the simultaneous measurement system of multipoint mechanical vibration for TWPs is utilized with a high-speed camera system based on both Python and open source computer vision (OpenCV) libraries. The proposed method has some advantages. First of all, the proposed method can realize simultaneous multipoint vibration measurements of TWPs. Secondly, the size of the template is decided by the detected interesting object which provide contextual cues for optical flow estimation. Finally, the motion information of the object of interesting or specific multipoint inside the template can be extracted with proposed method.
Three experimental studies validated the performance of the proposed system. The vibration measurement accuracy and capacity of the proposed system was first evaluated through a shaker vibration measurement experiment. The vibration measurement effectiveness and capacity for TWPs of the proposed system is then evaluated through a simultaneous multipoint vibration measurement experiment of an aircraft T tailplane model. The results using the proposed system achieved excellent agreement with those of traditional accelerometer sensors.
The changes in shading, lighting, and background conditions in the field made it difficult to extract accurate vibration from video images using the proposed system, further work will be carried out to improve the robustness of the proposed method to brightness variations.

Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.

Data Availability Statement
The data used to support the findings of this study were supplied by Jigang Wu under license and so cannot be made freely available. Requests for access to these data should be made to Jigang Wu by jwu@cvm.ac.cn.