Chatter has become the mainly limiting factor in the development of rapid and stable machining of machine tools, which seriously impacts on surface quality and dimensional accuracy of the finished workpiece. In this paper, a novel method of chatter recognition was proposed based on the combination of wavelet packet transform (WPT) and PSO-SVM in milling. The collected vibration signal was pre-processed by wavelet packet transform (WPT), and the wavelet packets with rich chatter information were selected and reconstructed. The selected wavelet packets can reduce the redundant noise and useless information. a combination of 10 time-domain and 4 frequency-domain feature parameters were obtained through calculating the reconstructed vibration signals. Compared to three methods of k-fold cross validation (k-CV), genetic algorithm (GA) and particle swarm optimization (PSO) to optimize the input parameters of SVM, the experiment results were shown that the PSO algorithm has is characterized by high accuracy. The proposed approach can recognize the stable, chatter and transition states more accurately than the other traditional approaches.