Determining reasonable tolerance patterns requires much knowledge and experience to balance the conflicting requirements. In previous studies, the process of tolerance specification has been based on the engineer’s experience and empirical data handed down from earlier projects. This paper proposes a learning model of tolerance specification of shaft part using deep learning architecture named CNN-BiLSTM-CRF, which is a combination of a Convolutional Neural Network, BiDirection Long Short-Term Memory network, and Conditional Random Field model. The CNN layer extracts the features from the CAD model; the Bi-LSTM layer composes the nonlinear relationship between tolerance specification and the features embedded by the CNN layer; the CRF layer predicts the tolerance sequence of the whole shaft part by jointly considering relations between neighbor tolerances. The proposed method directly takes the CAD model as the input and learns the tolerance specification from previous successful cases. Therefore, it does not require engineers to have any empirical knowledge.