The surface topography of the work-piece after machining has an important influence on the service performance, and the passive vibration of the tool will occur due to the influence of the milling force during the machining process. This paper focuses on the influence of the milling parameters and the passive vibration of the tool on the formation process of the surface topography. Firstly, the forming mechanism of the surface topography when the tool is passively vibrated is studied, and the cutting edge motion trajectory model considering the milling parameters and the tool passive vibration is established; The influence of milling parameters on surface topography with or without tool passive vibration is analyzed through experiments and simulations; The prediction models of maximum height Sz and three-dimensional arithmetic mean height Sa of surface topography are established by least squares support vector machine. The improved PSO algorithm is used to search the optimal solution of the kernel width coefficient and regularization parameters in the least squares support vector machine, and the program of the improved PSO-LSSVM prediction model is compiled. The prediction model proposed by error verification can provide a theoretical basis for the selection of actual milling processing parameters.