In the existing automatic drilling and riveting system, 90% of the faults are caused by the stuck in the rivet feeding system. In order to improve the efficiency of automatic rivet feeding system and reduce failure, a new rivet feeding system based on industrial robot and visual servo was proposed. By installing industrial camera and grasping device at the end of the robot, the quality and location of rivets were detected via machine vision. Then, the robot was guided to grasp and transfer rivets of specific types. Thus, quality of rivets is ensured before delivering and the process of delivery is optimized. The system weighs 1.1t and covers an area of 1.57m², which effectively saves the area and load of the rivet feeding system in the automatic drilling and riveting system. Experimental results illustrate that the system can detect rivets with a precision of 0.1mm and the feeding efficiency reaches six rivets per minute, which meets the performance requirements of automatic riveting and drilling system.