To investigate the effect of clearance induced by manufacturing errors on the performance of carbon fiber composite bolted joints, a top-down multi-scale analysis method was proposed. First, a three-dimensional progressive damage model was established to analyze the macroscopic mechanical response and damage mechanisms of the joint. The macroscopic strain was transferred to the representative volume element (RVE) model, where the crack initiation and propagation process at the microscopic scale were further explored. The accuracy of the numerical model was validated through five sets of experimental data. The results reveal a nonlinear relationship between the clearance amount and joint bearing strength. The bearing strength decreases most significantly when the clearance amount is 1.56%, with a reduction of 7.9%. Moreover, the clearance fit leads to the uneven stress distribution around the hole, which significantly reduces the joint stiffness. When the clearance amount is 3.08%, the joint stiffness decreases by 22.1%. The study also finds that matrix cracks initiate mainly in areas with dense fiber, while regions with high resin content tend to deterring the crack propagation. The results provide valuable theoretical and experimental guidance for the design and assembly of composite bolted joints.