In the task of hand-drawn sketch recognition, traditional deep learning methods have the insufficient of feature extraction and low recognition rate. To improve the insufficient, a novel algorithm based on double channel convolution neural network is proposed. First of all, the hand-drawn sketch is preprocessed to get a smooth sketch. And the contour extraction algorithm is adopted to get the contour of the sketch. The sketch and its contour are then used as input images of the CNN respectively. Finally, through performing feature fusion at the full connection layer, the classification results are obtained using the softmax classifier. The experimental results show that the proposed method can effectively improve the recognition rate of hand-drawn sketch.