In industrial vision system, metal surface is anisotropic under light in all directions and it is inevitable to cause local overexposure due to the natural reflection of active strong light, especially on the cylindrical metal surface. In this paper, injector valve is taken as the representative of cylindrical metal workpieces. Since the variety and complexity of cylindrical metal workpieces defects, and its contrast with the background of workpieces fluctuates, making samples annotating time-consuming and be of high cost. In order to solve the above challenges, this paper proposes an end-to-end weakly supervised learning framework to classify and segment defects. Firstly, a deep integrated residual attention convolutional neural network (IRA-CNN) is designed. IRA-CNN is composed of multiple IRA-Block. Two residual maps are included in IRA-Block to improve its bilateral nonlinearity and the robustness. IRA-block adds integrated attention module (IAM) which includes channel attention submodule and spatial attention submodule. The channel attention submodule adaptively extracts information from the global average pool layer and the global maximum pool layer to obtain the channel attention feature map. IAM can be well integrated into the IRA-CNN makes the neural network suppress the interference of useless background area and highlight the defect area. Finally, the weakly supervised segmentation method relies on Grad-CAM++ to generate saliency map to improve segmentation accuracy. The experimental results show that the accuracy of defect classification reaches 97.7% and the segmentation accuracy is significantly improved compared with the benchmark method in the injector valve dataset which include 6747 images.