A fretting wear model of rough surface that conforms to the actual situation is established to accurately reveal the wear mechanism of the connection structure. In the ABAQUS software, the UMESHMOTION subroutine and the energy dissipation model are used to simulate the fretting wear of double rough surfaces. The new model, a single rough surface model, and a smooth model are compared to analyze the differences between them. In addition, the influence of surface roughness, material, and friction coefficient on the fretting wear of rough surfaces is systematically explored through finite element simulation. The results show that the reliability of the model has been verified through Hertz’s theory and experiments. The stress and wear of the contact surface are more realistically reflected by the double roughness model. Besides, with the increase of surface roughness and material rigidity and the decrease of friction coefficient, the wear of the double rough surface model becomes more severe. The research work provides a theoretical basis for the design and performance prediction of the connection structure.