In this paper, similar and dissimilar 2A12 and 6061 aluminum alloy sheets are joined validly by self-piercing riveting, and a quasi-static experiment is performed to investigate the mechanical behaviors, failure modes and mechanisms of the joints. Also, a method based on deep learning algorithm to detect the appearance defects of the joints is proposed. The results show that the joints with similar 2A12 sheets contained the best static strength, the joints with similar 6061 sheets had superior anti-vibration performance, the joints with 6061-2A12 sheets presented the most decent comprehensive mechanical properties. The main failure mode of joints with similar 2A12 sheets was substrate fractured; the main failure mode of the other joints was pulled-out, and some of them with button-off. The fracture of 2A12 substrate belongs to the composite intergranular and microporous aggregate fracture. The method effectiveness was verified by experiments, and method detection accuracy could reach about 90%, and the detection speed was as high as 50FPS, which can effectively solve the problem that riveting quality was difficult to monitor.