The wide variety of components and high degree of customization used in the aerospace industry make it difficult to develop positioning fixtures. Visual localization technology is a key part of intelligent manufacturing, which is based on machine vision to determine the position of the workpiece. It does not require a positioning fixture, and can be widely used in a wide variety of work conditions. However, the generality of common visual localization algorithms is not very high. Algorithms are usually only used to detect specific objects. In this paper, a novel visual localization algorithm based on YOLOv5s object detection network and Siamese network (YOLO–Siamese change detection network) was proposed. The network introduced the ConvDiff (Convolutional Difference) module to improve the effect of the feature extraction in the change detection network, and a semi-supervised learning method was used to train the model. Experiments show that without using the target artifact dataset, the algorithm reached 99.3% of the AP@0.5, 89.6% AP@0.5:0.95 on the validation set, and the single frame inference time was 16.13 ms. Without requiring target artifact data, the proposed algorithm achieved high localization accuracy and fast operation speed, thus improving the robustness and versatility of visual localization algorithms.