In the five-axis flank milling process, it is difficult to calculate the instantaneous chip thickness due to the complex change of tool axis attitude. In order to improve the prediction accuracy of milling force, Firstly, the cutter tooth trajectory of cylindrical cutter is described by the geometric model of five axis flank milling. After establishing the micro element cutting force model, a calculation method of instantaneous cutting thickness is proposed, which is to calculate the mapping distance between the cutting point on the cutting edge and the rotary cylinder surface of the previous cutter teeth. Under the homogeneous coordinate transformation, the intersection calculation of line and plane in space is transformed into the intersection calculation of line and arc in two-dimensional plane. After considering the influence of tool runout on arc trajectory, the instantaneous chip thickness value is obtained by solving the equations. Finally, the milling force experiment was carried out on a five-axis machine tool. By comparing the measured force data, the simulation results are in good agreement with the measured values both in trend and size, which verifies the effectiveness of the established cutting force prediction model.