This article proposed two automatic methods to measure the NSA, as shown in Fig. 1. First, we performed the 3D reconstruction of the femoral model from the original CT data to obtain the 3D femoral point cloud data. Then, we used the PointNet++ network to segment the femur into the femoral head, femoral neck, and femoral shaft and applied mathematical features to fit the femoral neck axis and femoral shaft axis. Finally, we measured the NSA in 2D and 3D, respectively. In the 2D method, we found the optimal projection plane to simulate the hip anteroposterior radiograph, projected the segmented models onto the plane, and used mathematical features to fit the femoral neck and shaft axes. While in the 3D method, we used directly mathematical features to fit the femoral neck and shaft axes and calculated the NSA. Also, we performed manual measurements of the NSA and compared the results of these three methods.
Data collection
The CT data of 310 subjects from the Second Affiliated Hospital of Xi'an Jiaotong University were collected retrospectively from January 2017 to May 2021. Using a CT scanner (GE revolution CT, General Electric Company, Milwaukee, Wis), we collected CT datasets with 0.625*0.625 mm voxels and reconstruction thickness 1 mm. Only the intact femurs were included, and any diseases affecting the femur shape such as fracture, bone defect, bone tumor were excluded. The investigation followed the guidelines of the Helsinki Convention. We used the regional growth and mesh reconstruction steps in Mimics software (Materialise Inc., Leuven, Belgium) to perform the 3D reconstruction of the femur. And each reconstructed femur model was exported in the point cloud data format.
Femur segmentation based on PointNet++
PointNet++ is a neural network model proposed by Charles et al. in 2017[15], which is mainly used for point cloud segmentation and classification. This study used the segmentation module of PointNet++ to segment the femoral head, femoral neck, and femoral shaft. We divided the point cloud datasets into two parts: 50 femur data segmented manually by doctor were used to train the neural network, and the data of the remaining 260 femurs were used as test data. Cross-validation was adopted in the training, with 80% of the training data as the training set and 20% as the verification set.
Point cloud data can be expressed as n*3 matrix and has the characteristics of the disorder, which means that the arrangement order of each point in the point cloud does not affect its representation of the overall structure and shape in space. Most of the processing methods of the point cloud are to map the data into a two-dimensional image group or rasterize to perform subsequent processing. Still, the processing method often causes complicated calculations. The PointNet neural network model can take the point cloud data directly as the network's input and extract the point cloud features, which is a point-cloud-based 3D deep learning network proposed by Charles et al. at Stanford University in 2017[16]. The PointNet network model uses s a single max-pooling operation to aggregate the whole point set. Since the output of the symmetric function does not change with the order of the input factors, all points in the point cloud set can be processed by the symmetric function to obtain the global characteristics of the point cloud set. However, when the local features are processed, and the point cloud density varies from close to far, the processing effect of PointNet is not ideal. Charles et al. proposed a new neural network model based on PointNet and named it PointNet++ network to solve this problem[15]. The basis of the PointNet++ network is to divide the input point cloud into several overlapping regions at a certain distance in the metric space, extract the local features of the point cloud from each region, capture the geometric information, and repeat the whole process to get the feature set of the entire point cloud model. This process is similar to the convolutional neural network, which selects a specific convolution kernel size to extract the image's local features.
PointNet++, as a subsequent modification of PointNet, is essentially a layered version of PointNet, which can be used for point cloud segmentation and classification. This study used the segmentation network of PointNet++, as shown in Fig. 2. The segmentation process is to extract a global feature from the point cloud and gradually up-sample through this global feature. When up-sampling, Broadcasting replication is adopted, which means that the characteristics of the points near each point become the same as this point. However, this method will make it impossible to deal with some conflicting points or points that are not covered by the range. To achieve a better classification, PointNet++ uses the method of linear variance, and the specific formula is as follows:
The training environment of this model was as follows: python 3.6, pytorch1.1.0, cuda9.0, on a DGX-1 server with 8 Tesla V100 GPUs. The Adam optimization method is used for gradient descent (β1=0.9, β2=0.99), the learning rate is 0.01, batch size=128, epoch = 1000, and numpoint = 2048.
The cross entropy loss function we used is as follows:
Where M is the number of categories, \({\text{y}}_{\text{i}\text{c}}\) represents whether the predicted value is correct, and \({\text{p}}_{\text{i}\text{c}}\) represents the predicted probability that the sample i belongs to category c.
2D measurement of the NSA
Projection
According to the requirements of the hip anteroposterior radiograph, internal rotation of the lower limb counterbalances the effect of the femoral anteversion and keeps the femoral neck axis the longest. The purpose of our 2D method was to simulate the standard position of the hip anteroposterior radiograph. We used the following method to determine the projection plane: first, we segmented the femoral head, femoral neck, and femoral shaft using the Pointnet++ network and extracted 10% of the distal part of the femoral shaft. Next, as a sphere-like object, the femoral head was fitted to a spherical surface by the least square method to obtain the center coordinates. The calculation formula is as follows:
The centroids of the femoral neck and 10% of the distal femur were also calculated. Finally, the three points were used to determine the projection plane where the 2D automatic measurement of the NSA was performed. It is shown in Fig. 3. The projection principle of this method used three dimensional plane equations in space as follows:
The vertical foot of any point Vi in the point cloud to the plane is denoted as Vi '(x,y,z), and the line ViVi' is parallel to the normal vector of the plane. The parametric equation of line ViVi' is as follows:
Femoral neck axis fitting
This study used the PointNet++ network to segment the femoral regions and projected a femoral model to the projection plane. Next, we used Random sample consensus (RANSAC) to obtain the center of the sphere of the projected femoral head and the centroid of the femoral neck, which formed the femoral neck axis[17]. The basic process of the RANSAC is as follows:
-
The minimum data set that could estimate the model was randomly selected from the input data, and the corresponding model parameters were calculated using the data set.
-
We Substituted all the data into the model and calculated outliers and inliers. Data exceeding the error threshold were considered outliers, while data less than the error threshold was considered inliers.
-
The number of repetitions of steps 1-2 was related to the outliers proportion and confidence of the model. The calculation process is shown in Equation (8).
S is the number of minimum tests required, P is the confidence, p is the percentage of inliers, and K is the number of random samples.
Due to the irregular shape of the femoral neck, we solved the centroid of the femoral neck, and its formula is as follows:
Where\(\text{i} = 1,....,\text{C}\)
Femoral shaft fitting
We used the projected femoral shaft and connected the centroid of 60%~ 70% of the total length of the femoral shaft and the centroid of 70%~80% of that as the femoral shaft axis.
Calculation of the NSA
While obtaining the femoral neck axis and the femoral shaft axis, we used the 3D vector angle formula to calculate the NSA. The calculation formula is as follows:
Where A is the vector representing the femoral neck axis, and B is vector representing the femoral shaft axis.
3D measurement of the NSA
We connected the spherical center of the femoral head and the centroid of the femoral neck obtained by the above steps as the femoral neck axis. We defined the line in space connecting centroid of 60%~ 70% of the total length of the femoral shaft and the centroid of 70%~80% of that as the femoral shaft axis. We finally calculated the NSA using the 3D vector angle formula same as the 2D method.
In terms of the software in 2D and 3D methods, we used the Visual Studio (version 2015, Microsoft, Inc, Seattle, Washington, USA) and Point Cloud Library (version 1.8.1) to calculate the NSA, and applied cloud compare (version 2.11, alpha, Anoia) to implement visualization.
Manual measurement of the NSA
We used the Solidworks software (version 2017, Dassault Systemes Company, Concord, 121 Massachusetts) to achieve the manual measurement. Firstly, we imported the femoral model and adjusted the position of the model to observe from the femoral head to knee with the overlap of the greater trochanter and the posterolateral condyle of the knee and the horizontal placement of the femoral neck axis. Next, we generated the plane 4 parallel to the screen, drew the femoral neck axis on the 2D plane 4, and created the new plane 5 perpendicular to the plane 4 with passing the femoral neck axis. Finally, we achieved the measurement on the 2D plane 5. We defined the femoral neck axis as the line connecting the center of the circle fitting the femoral head with the midpoint of the transverse axis of the femoral neck and the femoral shaft axis as the line connecting the two midpoints of the two transverse axes on the proximal femoral shaft. The NSA was the intersection angle between the two axes, which was shown in Fig. 4.
Evaluation of the results
To accurately evaluate the segmentation effect of this algorithm on the femoral regions, The Dice coefficient and MIoU index are used to evaluate the performance of the segmentation. The calculation formula is as follows:
TP: the prediction is correct, the prediction result is positive, and the truth is positive. FP: the prediction is false, the prediction result is positive, and the truth is negative. FN: the prediction is false, the prediction result is negative, and the truth is positive. TN: the prediction is correct, the prediction result is negative, and the truth is negative. X is the segmentation result set output by the algorithm, and Y is the segmentation result set manually segmented by the doctor.
To evaluate the accuracy of the two automatic methods and analyze the consistence between the 2D and 3D methods, we calculated the error and average accuracy. The formulas are calculated as follows:
Also, we used the two-sample Student’s t-test to determine the significance of the difference between any two of three methods. All statistical analyses including the Dice coefficient, MIoU index, average error, average accuracy, and t test were completed in Python version 3.6. And P values less than 0.05 were considered significant.