Self-piercing riveting technology is suitable for joining dissimilar materials such as aluminum and steel, and the joint performance is reliable, so it has a wide application scenario in the aviation industry. However, there are few relevant researches on nondestructive detection of self-piercing riveting defect at present. This paper proposes a deep learning-based partial riveting defect detection algorithm for self-piercing riveting. Firstly, the mechanical properties of partial riveting self-piercing riveting parts decreased by 5.6% compared with normal riveting parts through shear mechanical properties test. Then, the degree of partial riveting was defined in the range of 0 – 10 by the external features of self-piercing partial riveting parts. Finally, the detection algorithm was studied, and the detection effect difference between single-step detection and two-step detection was explored. The detection scheme of YOLOv5s (You Only Look Once v5s) and ResNet18 was proposed. In addition, the Gradient-weighted Class Activation Mapping (Grad-CAM) was used to visually explain the differences in the effects of different detection schemes. The test results showed that the proposed detection scheme of YOLOv5s plus ResNet18 could achieve 100% accuracy in the collected data test set, which was higher than the 95.18% accuracy achieved by only using YOLOv5s, and much higher than the 84.1% accuracy achieved by only using ResNet18.