Hand pose estimation is the basis of dynamic gesture recognition. In vision-based hand pose estimation, the joints of the human hand are highly flexible, and problems such as local similarity and severe occlusion have great influence on the estimation of hand posture. In order to identify the complicated hand posture, the structural relationship between the hand nodes is established, more accurate hand pose estimation can be achieved through the improved Nonparametric Structure Regularization Machine (NSRM) in this paper. Based on the NSRM network, the backbone network is replaced by New High-Resolution Net (NHRNet), then the input and output channels of some convolutional layers are reduced. Finally, a public dataset is used to conduct the hand pose estimation experiments. The experimental results show that the optimized NSRM network has higher accuracy and faster recognition speed for hand pose estimation.