Carbon fiber reinforced plastic (CFRP)/TC4 stacks are widely used in aerospace fields. Drilling of CFRP/TC4 stacks is generally performed using a single-process method. In this study, ultrasonic-assisted drilling and pecking drilling were combined to form a novel ultrasonic pecking drilling (UPD) process for drilling these materials. The variation in drilling sequences for stacks significantly affects hole quality and tool wear. Therefore, this paper analyzes the thrust force, CFRP exit damage, and tool wear during UPD of CFRP/TC4 stacks. The results indicate that the drilling in the CFRP→TC4 direction reduces CFRP exit damage, with the maximum delamination factor decreased by 21.9%; Drilling in the TC4→CFRP direction generates lower thrust force and causes less wear on the tool’s main cutting edge; Scanning electron microscopy (SEM) observations and surface roughness measurements of the hole walls under both drilling sequences demonstrate that titanium chip entanglement in the TC4→CFRP direction improves hole wall quality.