With the rapid development of modern medical science and technology, medical image classification has become a more and more challenging problem. However, in most traditional classification methods, image feature extraction is difficult, and the accuracy of classifier needs to be improved. Therefore, this paper proposes a high-accuracy medical image classification method based on deep learning, which is called hybrid CQ-SVM. Specifically, we combine the advantages of convolutional neural network (CNN) and support vector machine (SVM), and integrate the novel hybrid model. In our scheme, quantum-behaved particle swarm optimization algorithm (QPSO) is adopted to set its parameters automatically for solving the SVM parameter setting problem, CNN works as a trainable feature extractor and SVM optimized by QPSO performs as a trainable classifier. This method can automatically extract features from original medical images and generate predictions. The experimental results show that this method can extract better medical image features, and achieve higher classification accuracy.