Aiming at the problem that the data samples are small and nonlinear in the modeling of geometrical error items of CNC machine tools, the SVR (support vector regression) with unique advantages in the nonlinear regression analysis of small sample data sets is studied, and based on which the geometric error prediction model of CNC machine tools is established. This paper analyzes the problems of the difficulty of measuring points and the calculation of cumulative error in the nine-line method commonly used in the detection of geometric error, and then proposes an improved method to increase the measurement of the straightness of each measurement line and the calculation model of the correction error term. The Gaussian Radial basis kernel function is chosen as the kernel function of the SVR model, and the cross-validation method is used to select the appropriate model parameters to solve the convex quadratic programming problem, and then the geometric error prediction model is established. Taking the X-axis of the QLM27100–5X five-axis gantry machine as an example, the geometric error sample data is obtained by measuring and identifying based on the improved nine-line method, and then the geometric error item prediction model is established based on the support vector regression machine and the least squares method respectively, and the prediction accuracy of the two models is compared. The results show that the predictive MSE of the former is 0.0238, which is less than 0.072 of the latter. It proves that the support vector regression model has higher predictive accuracy in small sample set.