Fruit cracking and rust spot are common diseases of nectarine which would seriously affect its yield and quality. Therefore, it is essential to construct disease detection models for agricultural products with high speed and accuracy. In this paper, a sparse dictionary learning method was proposed to realize the rapid and nondestructive detection of nectarine disease based on the multiple color features combined with the improved K-SVD (K-Singular Value Decomposition). According to the color characteristics of nectarine itself and the significant color differences existing in the three categories of nectarine (diseased, normal and background parts), multiple color spaces of RGB, HSV, Lab and YCbCr were studied. It was concluded that the G channel in RGB space, Y channel in YCbCr space and L channel in Lab space can better distinguish the diseased part from the other parts. At the model training stage, pixels of the diseased, normal and background parts in the nectarine image were randomly selected as the initial training sets, then the neighborhood image block of the pixels were selected to construct the feature vectors based on the above color space channels. An improved LK-SVD (Label K-SVD) dictionary learning algorithm was proposed that integrating the category label into the process of dictionary learning, then an over-complete feature dictionary with significant discrimination was obtained. At the model testing stage, orthogonal matching pursuit (OMP) algorithm was used to sparse reconstruction the original data which can obtain the classification categories based on the optimized feature dictionary. Experimental results show that the sparse dictionary learning method based on multi-color features combined with improved LK-SVD can detect fruit cracking and rust spot diseases of nectarine quickly and accurately, and the average overall classification accuracies were 92.06% and 88.98% respectively, which was better than k-nearest neighbor (KNN) and support vector machine (SVM). It is demonstrated that this study can provide technical support for the diseases detection of agricultural products.