Rice is one of the most important staple crops in the world and provides millions of people with a significant source of food and income. Problems related to rice classification and quality detection can significantly impact the profitability and sustainability of rice cultivation, which is why the importance of solving these problems cannot be overstated. By improving the classification and quality detection techniques, it can be ensured the safety and quality of rice crops, and improving the productivity and profitability of rice cultivation. However, such techniques are often limited in their ability to accurately classify rice grains due to various factors such as lighting conditions, background, and image quality. To overcome these limitations a deep learning-based classification algorithm is introduced in this paper that combines the power of convolutional neural network (CNN) and long short-term memory (LSTM) networks to better represent the structural content of different types of rice grains. This hybrid model, called CNN-LSTM, combines the benefits of both neural networks to enable more effective and accurate classification of rice grains. Three scenarios are demonstrated in this paper include, CNN, CNN in combination with transfer learning technique (CNN-TL), and CNN-LSTM deep model. Combining CNN and transfer learning techniques can improve the accuracy and performance of classification models by reducing overfitting, reducing training time and cost, and enabling better representation of complex data structures. The performance of the mentioned scenarios is compared with the other deep learning models and dictionary learning-based classifiers. The experimental results demonstrate that the proposed algorithm accurately detects different rice varieties with an impressive accuracy rate of over 99.85%. The algorithm also displayed high accuracy in identifying quality for varying combinations of rice varieties with an average accuracy of 99.18%. These results indicate that the proposed algorithm is a promising approach for improving the accuracy and efficiency of rice classification and quality detection.