Accurately solve the cutter–workpiece engagement (CWE) of ball-end milling has become a key problem for its mechanical behavior. In view of the limitations of the existent CWE solution methods of ball-end milling, such as low efficiency, poor universality and weak engineering applicability, a CWE solution method based on homogeneous transformation matrix is proposed to realize its prediction and analysis. Firstly, in light of the analysis of geometric characteristics in milling process, the analytical equations of vertical CWE are deduced; Then, from the perspective of multi-body system theory, the milling posture parameters of ball-end milling are characterized as coordinate matrix by homogeneous transformation method, and its analytical expression under any posture is derived. Besides, the milling process is discretized in time domain to analyze the effective milling edge and its evolution process in CWE. On this basis, the experiment for ball-end milling force is carried out to verify the reliability and accuracy of the solution method and milling force analysis. The results show that the proposed method can simplify the solution process of CWE, and the prediction of milling force is reliable and accurate; The increase of rake angle or roll angle will reduce the milling time of cutting edge and reduce the average milling force in the milling cycle; A larger milling force can be obtained with a smaller rake angle.