3.1 Camera parameter calibration
As shown in Fig. 2, the camera imaging process can be expressed as the conversion of image points in the world coordinate system, camera coordinate system, retinal coordinate system, and pixel coordinate system[10, 11].
It can be seen from Fig. 2 that a point Pw in the world coordinate system needs to be transformed by 4 coordinate systems before it can be represented by pixel coordinates in the imaging plane. According to the principle of homogeneous coordinate transformation, the mapping relationship between the pixel coordinate system and the world coordinate system can be obtained, as shown in formula (1):
(1)
A is the internal parameter matrix of camera, which is determined by the camera's internal properties. M is the external parameter matrix of camera, which describes the pose relationship between the camera coordinate system and the world coordinate system.
Through the calibration algorithm that was put forward by Zhang Zhengyou[12], the corresponding internal and external parameter matrix can be obtained by using a two-dimensional checkerboard calibration board. In this paper, the camera is calibrated by the MATLAB camera calibration tool, and the internal parameters of the camera are shown in Table 4.
Table 4
Camera internal reference
fx
|
fy
|
u0
|
v0
|
1806.276
|
1805.866
|
313.818
|
218.459
|
3.2 Hand-eye calibration
The most important step of the combined application of machine vision on industrial robots is hand-eye calibration. The purpose of hand-eye calibration is to realize the transformation of objects in the world coordinate system and the robot coordinate system, so that the robot can perform the corresponding pose transformation according to the image information.
Through the Tsai-Lenz hand-eye calibration algorithm, the hand-eye relationship matrix
of the visual system of "Eye-in-Hand" type[13] is obtained:
(2)
The hand-eye relationship matrix represents the transformation matrix from the camera coordinate system to the robot end coordinate system.
3.3 Image feature extraction and positioning
The visual positioning of the system has two research objects, namely the goniometer and the laser sensor spot. The outer contours of the two can be approximated as circles with diameters of 9 mm and 0.8 mm respectively. The positioning images of both research objects are shown in Fig. 4.
Since the center of the laser spot has a high brightness and is quite different from the surrounding environment, the spot image is extracted according to the color characteristics. Compared with other image visual features, color features are less dependent on image orientation, viewing angle, size and other conditions, so they have higher robustness. First, the HSV color threshold of the laser spot center is determined. Set the upper and lower limits of the threshold by extracting the HSV parameters of the pixel points in the center of the light spot to (0,0,240), (170,20,255), respectively, and traverse the pixels of the entire image. The pixel values that are less than lower limit or greater than upper limit of the threshold are set to 0, and the pixels values in the threshold area are set to 255, and the ROI mask of the image can be obtained.
The mask and the original image are combined to extract the laser spot image, as shown in Fig. 3(a). There will be some noise in the obtained laser spot feature image. In order to eliminate the noise, median filtering was performed on the image to obtain a smoother spot image, as shown in Fig. 3(b).
After the positioning images of laser sensor spot are obtained, it is necessary to obtain the positioning images of goniometer. Using the minimum closed circle fitting algorithm and Hough circle detection[14, 15] algorithm of opencv, the laser spot and the goniometer image can be delineated by the closed circle, and the pixel coordinate position of the circle center can be determined as shown in Fig. 3(c) and Fig. 3(d):
After obtaining the pixel coordinates of the center of the circle, according to formula 1, the coordinates of the corresponding laser spot and goniometer in the camera coordinate system can be solved, and the coordinates in the tool coordinate system of mechanical arm can be obtained through the hand-eye relationship matrix (as shown in formula 2), so as to achieve coordinate positioning.
Three sets of visual positioning experiments are carried out on the goniometer to determine its positioning error. The real coordinate values come from the coordinate position of manual calibration. The experimental results are shown in Table 5:
Table 5
Experimental error of visual positioning of goniometer center
number
|
positioning coordinate
/(mm)
|
real coordinate
/(mm)
|
positioning error /(mm)
|
X
|
Y
|
X
|
Y
|
X
|
Y
|
1
|
812.92
|
335.66
|
811.16
|
334.23
|
1.76
|
1.43
|
2
|
802.03
|
327.88
|
800.44
|
326.25
|
1.59
|
1.73
|
3
|
820.14
|
342.83
|
818.46
|
341.06
|
1.68
|
1.77
|
It can be seen from Table 5 that the average error of the center coordinates of the goniometer in the X and Y directions only by monocular vision positioning are 1.68 mm and 1.61 mm respectively, which far exceed the error range allowed by the position calibration of the above sample points.