This research presents an essential solution for classifying ultrasound diagnostic images describing seven types of ovarian cysts: Follicular cyst, Hemorrhagic cyst, Corpus luteum cyst, Polycystic-appearing ovary, endometriosis cysts, Dermoid cyst, and Teratoma. This work proposed a novel technique using images of ovarian ultrasound cysts from an ongoing database with this motivation. Initially, the work is followed by removing noise in preprocessing, feature extraction, and finally classifying using new Deep Q-Network with Harris Hawks Optimization (HHO) classifier. Automatic feature extraction is implemented using the recent popular convolutional neural network (CNN) technique that extracts image features as conditions in the reinforcement learning algorithm. With this, through the procedure of a new deep Q-learning algorithm, Deep Q-Network (DQN) is generated to train a Q-network. The swarm-based method of HHO utilized the optimization method to produce optimal hyperparameters in the DQN model known as HHO-DQN, a novel technique for classifying ovarian cysts. Extensive experimental evaluations on datasets show that the proposed HHO- DQN approach outperforms existing active learning approaches for ovarian cyst classification. Compared with the ANN, CNN, and AlexNet models, the performance of the proposed model is better in terms of precision, f-measure, recall, accuracy, and IoU. The proposed model has achieved 96% precision, 96.5% f-measure, 96% recall, 97% accuracy, and 0.65 IoU.