Ovarian Cysts Classi�cation Using Novel Deep Q-Learning With Harris Hawks Optimization Method

: This research presents an essential solution for classifying ultrasound diagnostic images describing seven types of ovarian cysts: Follicular cyst, Hemorrhagic cyst, Corpus luteum cyst, Polycystic-appearing ovary, endometriosis cysts, Dermoid cyst, and Teratoma. This work proposed a novel technique using images of ovarian ultrasound cysts from an ongoing database with this motivation. Initially, the work is followed by removing noise in preprocessing, feature extraction, and finally classifying using new Deep Q-Network with Harris Hawks Optimization (HHO) classifier. Automatic feature extraction is implemented using the recent popular convolutional neural network (CNN) technique that extracts image features as conditions in the reinforcement learning algorithm. With this, through the procedure of a new deep Q-learning algorithm, Deep Q-Network (DQN) is generated to train a Q-network. The swarm-based method of HHO utilized the optimization method to produce optimal hyperparameters in the DQN model known as HHO-DQN, a novel technique for classifying ovarian cysts. Extensive experimental evaluations on datasets show that the proposed HHO-DQN approach outperforms existing active learning approaches for ovarian cyst classification. Compared with the ANN, CNN, and AlexNet models, the performance of the proposed model is better in terms of precision, f-measure, recall, accuracy, and IoU. The proposed model has achieved 96% precision, 96.5% f-measure, 96% recall, 97% accuracy, and 0.65 IoU.


INTRODUCTION
Doctors are using transvaginal ultrasound to diagnose some common ovarian cysts in postmenopausal women (TVU). The regular monitoring of apparent, simple cysts in postmenopausal women seems uncertain when ovarian cysts' genetic basis is unidentified [1]. This research examines how and when to accurately describe and interpret ultrasound images to develop a computer-aided diagnostic for ultrasound ovarian abnormalities. Rapid advancements in imaging technology have rendered manipulating patient images during clinical diagnosis especially easy. Capturing and metrics labelled related cases from a detailed database is a crucial problem of rising significance as the number of images is increasingly growing. Creating and testing feature extraction and classification frameworks for medical images have frequently been used as one of the primary research domains in high-impact possibilities [2]. These methods have contributed significantly to the medical field examined as one of the primary research disciplines of image extraction and classification in high-impact opportunities. MRI images of the heart and brain, Radiology images, CT images of the liver, X-ray images, dermatology images, and ultrasound images of the kidney and breast have been subjected to content-based retrieval classification algorithms [3]. No new studies have ever discussed image recognition techniques on ultrasound images of ovarian defects, which aims to contribute. The accurate characterization of ovarian ultrasound images is crucial for diagnosing, but it did not prove easy due to the images' fundamental diversity.
In contrast, visual ultrasound analysis of endogenous phenomena remained the key to determining their severity. For this purpose, digital ultrasound testing is now commonly recognized and practised as the most widely accepted and performed diagnostic modality for non-invasive evaluation of ovarian cysts and other forms of ovarian abnormalities [4], prompting the development of ultrasound-based algorithms. This process, however, is heavily dependent on functional experience in identifying morphology, including features of different types of ovarian abnormalities in their ultrasound images. As a result, novice ultrasound operators often have difficulty distinguishing between various cysts, resulting in a lower percentage of positive diagnoses [17][18].
A computer-based system for ultrasound image recognition is used to aid in diagnosing ovarian anomalies [5]. Even if the incorrect diagnosis leads to unwanted benign cysts or missed cases in the worst-case scenario, supportive services may help them improve their diagnostic accuracy provided by inexperienced technicians. With that kind of encouragement, the active learning mechanism was used to identify ovarian cysts as a reinforcement learning challenge, which was perceived as incremental sampling via the labelling process. Active learning is a structured decision-making process in which a reinforcement learning agent gathers information by interacting with the situation and then makes several decisions to achieve a goal. Regardless of the actual findings, including the CNN model's feature vectors, the RL agent should decide whether or not to annotate the image. The deep Q-network also signifies the RL agent [6]. As a consequence, as the ability of the classifier grows, the decision strategy shifts. In overview, the work's main contribution consists of three parts: • The Wiener filter is used to remove noise in the preprocessing stage when collecting images. • A modern active learning method was implemented, which uses the reinforcement learning algorithm to detect image classification standards automatically. HHO has also been used to improve DQN hyperparameters extracted from a DQN. • Experiments with different CNN models on image datasets show that using a reinforcement learning algorithm to classify the ovarian cyst for active learning results in performance improvements in progress methods.
The rest of the article is arranged along these lines: Section 2 discusses similar studies on ovarian image classification. Section 3 outlines the proposed HHO-DQN technique for particularly ovarian cyst classification. Section 4 discusses the findings and discussions. Section 5 describes the conclusion about the proposed model and future work.

RELATED WORK ON OVARIAN CLASSIFICATION
First, the need for a fast, accurate clinical differentiation between normal and abnormal ovarian masses is being addressed by researchers in connection with vaginal ultrasound to assess the safety and reliability of echo pattern classification. This research classified six ovarian cyst image sample types of 405 TVU ovarian masses anatomically treated from January 2011 to December 2012. This work compared subsequent classifications of patients who underwent histopathology diagnoses with the diagnostic precision of the carcinoma diagnostic diagnosis [7]. New studies show that many anomalies were historically considered primary ovarian tumors. Explicitly, in the fallopian tube, endometrial and clear cell carcinomas exist. It outlines the recent advances in molecular pathology, which have considerably improved their understanding of ovarian carcinomas biology and are critical for patient management in this analysis [8].
Recent research shows a morphologically dependent MRI-guided approach for the Ovarian Tumor Differential Diagnosis [9]. Premature ovarian tumors are divided into three groups based on endothelial cells, memorandum, and fibrous tissue tumors moderate, intermediate, or cancerous ovarian tumors. The adnexal masses were classified into four significant groupings using an imaging approach focused on morphological looks: desmoplastic cysts, adenomas of the cyst, endometriosis, and solid, the most common being endometriosis. The research architecture enables the standard VGG-16 model with the ultrasound images' data [10]. The VGG-16 architecture is a deep neural learning network with 16 layers trained on the ImageNet dataset. The last four layers of the VGG-16 network have been updated for network adjustment. One model will tell whether such ultrasound images show an ovarian cyst or not.
In this study, AlexNet based Deep Convolutional Neural Networks (DCNN) were used to automatically classify the different forms of cytologic ovarian cancers [11]. The classification of ovarian tumors based on ultrasound images has recently been investigated in deep learning. To begin by noticing an ultrasound image dataset containing 988 ovarian cyst image samples of three ovarian tumour forms, despite the lack of publicly available ultrasounds images. Secondly, compare the overall CNN models' overall capacity [12]. Many studies have used Artificial neural networks (ANN) to resolve the classification of ovarian cancer. Ultimately, physicians rely on the precise rating when making decisions [13], except that the procedure mentioned above is successful in the ovarian rating.

PROPOSED METHODOLOGY
The proposed DQN and HHO for ovarian cyst classification are described in this section, and Figure 1 depicts the proposed scheme's process. The ovary ultrasound image dataset is initially chosen and preprocessed with the Wiener filter to eliminate unnecessary noise and errors. The feature extraction was performed using CNN to improve classification accuracy. The features are then classified using DQN and HHO. Backpropagation is the DQN training method used with HHO. HHO has been used to optimize DQN hyperparameters, including the Discount factor (γ), Batch size, and dropout rate. The output results indicate that the proposed HHO-DQN outperformed existing ANN, CNN, and AlexNet.

Image collection
In progress, an ultrasound image database is being developed for different ovarian abnormalities. The database includes 478 ultrasound ovarian cyst images collected in the Royal Victoria Hospital in Montreal for diagnostic purposes. The images in the database were classified into seven ovarian cyst groups. This classification has been validated by many or even more domain experts by examining the corresponding known pathological diagnosis [3].

Image Pre-processing wiener filter
In the preprocessing section, the input image can be of varying size and comprise noise, which would be of varying shades. The noise is removed through filtering algorithms. Among many of the various filters, the wiener procedure is implemented [14]. Wiener filters are the most exemplary linear filter that encompasses the linear approximation of the anticipated signal arrangement on or after an additional predefined pattern that is not adaptive. A traditional Wiener filter is a convolution filter that utilizes just a frame to show the neighborhood's shape and size to be calculated. Filtering with more overly large Viennese masks with far more miniature masks and less blurry can be more effective. In a US ovarian image of size ( × ), each pixel must reflect a single stationary point's strength in front of the camera. The Wiener filter aims to remove noise that has distorted a signal that focused on statistical analysis. Typical filters are designed to achieve a specific dynamic range. The Wiener filter takes a particular method to filter and is distinguished by the following benefits:  Static linear random processes with proven optical characteristics are supposed to be the signal and the additive noise.  Essential requirement: a filter ought to be physically feasible.
In other words, a non-causal approach should be removed from that requirement.  This phase used Minimum Mean-Square Error (MMSE) as a performance measure To illustrate and transform to 2D, consider the following as in equation (1): Where * signifying convolution, signifies the unknown actual ovarian cyst image with × pixel value, signifies the impulse response of a linear, time-invariant filter, signifies incremental unfamiliar noise regardless of , and illustrates the observed image. After that, a deconvolution filter must be found to evaluate as follows in equation 2: where ̂ indicates the value of that reduces the mean square error.
The transition function of , in the frequency domain, is as given in equation 3, In which ( , ) is the Fourier transform of the probability mass equation, ( , ) is the Fourier transform of , PS is the Power spectrum of the signal process carried out to obtain the Fourier transform of the signal collinearity, and ( , ) is the power spectrum of the noise ( ) practice calculated through enchanting the Fourier transform of the noise autocorrelation. Reinforcement Learning (RL) system: RL is divided into three main components: state space, action space, and reward function. The RL agent aims to acquire a mapping function starting state space (SS) to action space (AS). The RL agent is inspired to act. The RL agent's goal is to maximize the accumulated rewards.

Feature Extraction using CNN
Here will now expand on each of the system's three components individually before implementing the whole structure.
State-space: The RL agent decides which way to proceed based on each stage's in-progress state. To make informed decisions, the government should provide comprehensive information to the RL agent. Here signifies the state with a continuous vector that comprises the extracted features with the use of CNN model, , as well as the prediction scores of the labelled ovarian cyst image samples denoted as ( ). The state-space signified by = { }, where = ( , ( )), besides the superscript signifies the state at a particular time.
Action space: In this stage, the action suggests whether the method is required to label the present ovarian cyst image sample, = {0,1}signifies the action space. When action = 1, the model is manually annotated, as well as the newly annotated images added to the labelled training collection. When there is no more data, or the annotation budget is exhausted, the active learning system uses a unique 'termination' framework to end the process.
Reward function: An essential part of the reinforcement learning algorithm is the reward function. It is used to assess the consistency of the RL agent's behaviour. The reward function is rewarded at the end of the session in most previous works, during which the expected output of the trained model on labelled data is measured. However, after a long game, it is difficult to attribute this delayed reward to individual behaviour.
In which signifies the i th unlabelled ovarian cysts image function that the CNN model hails out and is the penultimate layer's output. Mainly on the situation of = 0, define the reward function (RF) as given in equation 5, whereby ( , )corresponds to the entropy loss for the unlabelled ovarian cyst input image . As well as being an adjustable parameter that varies with the trained model's functionality. This incentive feature compels the model to annotate ovarian cyst image samples where it has less conviction. That was similar to the previous approaches focused on uncertainty, except that the selection criterion was applied. Since the model's capacity is relatively poor at the start of the training, set the smaller that suggests that the model's extracted features are less accurate. If the training procedure progresses and the amount of annotated ovarian cyst image samples increases, the value of keeps growing.

PROPOSED HHO-DQN METHOD
DQN algorithms are the most important in machine learning, yet they have made significant strides in many real-world problems in recent years. The design of the hyperparameters has a considerable impact on the network's accuracy for a specific mission. Acquiring the correct collection of hyperparameters is a time-consuming and expert-required process. To address this issue, implement the HHO methodology and suggest the HHO approach for hyperparameter optimization and structural analysis. The whole framework of HHO-DQN is illustrated in Fig. 4.
Deep-Q Network (DQN): Since defining the action space, state space, and reward function, the active learning algorithm based entirely on the RL system developed. The model shown here is close to the VGG-16 net model that begins with five convolutional blocks, such as two convolutional layers. The other three blocks each have three convolutional layers and an utterly connected layer that outputs a 52-dimensional feature vector. The design DQN divides into two branches: the Q-network branch, which reflects the RL agent. The Q-network branch is used for classification that is made up of two fully linked layers. The input consists of the function description and the stabilized softmax output ( ) from the classification branch. As well the Qperformance network's is a 2D vector, with 1D signifying action = 0 and the other signifying action = 1. According to the Q-network output, the RL agent will take action with the highest value.
Since the DQN now has a detailed summary of the processing system, the next problem is to train it to get a proper data collection criterion. Various factors select different data, resulting in different models of results. As discussed above, the system has two branches, one of which has to understand the status action-value function, and the other has a classification. Since both modules are following the same input, learning both can be considered a multi-task learning procedure at the same time.
The standard softmax layer was used to calculate the ultimate classification performance for classification and trained with the algorithm of back-propagation. Thus, throughout the following, concentrate on -network training. To begin, establish a tuple ( , , , +1 ) to describe an RL agent transfer. It implies the in-progress state that if the agent obtains a reward it while acting , the in-progress state transitions to the next state +1 .
One ovarian cyst image of active learning could yield of above tuple set. The deep Q-learning scheme [15] is utilized to study the data selection policy via map function ( , ) → , which evaluates the efficiency of taking from state . In deep Qlearning, the RL agent unexpected results Q based on the rewards obtained within each ovarian cyst image ( , ). Here signifies the target Q-value depending on the latest parameters . Following [15], the testing replay memory (RM) is often accustomed to preserve for each transition tuple ( , , , +1 ) that formed in an ovarian cyst image is a minibatch of modifications selected from the RM, and the also decreased. Until training the two network branches and together, the stochastic gradient descent method was used to update the Q-network parameters in mini-batches. The proposed framework was created using mini-batches in an endwise format because the BP algorithm could obtain all layer parameters simultaneously. The network's efficiency is strongly reliant on the specification of the hyperparameters. The most serious problem is there is no standard framework for deciding the optimal set of DQN's hyperparameters and developing the network's structure. Besides, no specific architecture can be generalized to any problem; each issue instance necessitates an independently constructed DQN architecture. The authors developed an automated approach for hyperparameter optimization and structure design using the HHO algorithm in this study to address this issue. In opposition to random search, the swarm-based algorithm learns with each execution and progresses toward better hyperparameter values with each subsequent iteration. HHO's fitness function: The DQN weight is optimized for error reduction, leading to improved image classification accuracy. At each iteration, the weight values are fine-tuned, and the network trained as represented in equation 8, Here is an abbreviation for Desired value, is an abbreviation for Predicted value, and is an abbreviation for several ovarian cyst images.
Harris Hawks Optimization: Harris hawks are known for their teamwork in tracing, encircling, approaching, and finally attacking rabbits' potential prey in most cases. In hunting evading preys, an inventive maneuver known as "surprise pounce" would be used appropriately. The implementation process of unexpected gang-up is as follows: team members launch active attacks from distinct ways but instead converge on the intended rabbit. A switching strategy-the chase would then continue under another team members in progress leader, to perplex the attempting to spurt rabbit whenever the best hawks appear to come towards its rabbit and loses sight of it. The phases of exploration and exploitation are included in the HHO algorithm [16]. The hawks will hang upside down at unexpected times to evaluate different areas and track and identify the rabbit, mainly during exploration. In the exploitation phase, hawks will affect having pounces, or rapid team jumps to manipulate the neighborhood of intended prey. Hawks' locations are suggested as sound systems. As per hawks' different approaches at different phases, HHO split into two ladders: seeking prey and hunting prey. The following are the specific implementation details of HHO. Seeking Prey: At this point, the Harris hawk will start checking a search space for quarry. This phase refers to the HHO algorithm's exploration process. Two unreachable destination schemes are used during the "seeking prey" phase to visualize the Harris hawks' waiting, monitoring, observation, and special operations actions even before prey makes it appear. The two compliance processes are as equal as possible and described as follows in equation 9: where is a definite in-progress iteration number, ( ) is the in-progress hawk position, and ( + 1) is the hawk position vector during the next iteration.
( ) signifies an individual in the hawks during the iterative process, ( ) signifies the average position vector of all associates in the inprogress population, and ( ) Signifies the prey's position that is expected to be the hawks' situation with the highest appropriateness value in each iteration . The superior plus inferior bounds of the decision variable for a specific problem are signified by and , respectively. A similar behaviour between 0 and 1 is predicted by 1 , 2 , 3 and 4 .
Period of transition: Although time passes, the likelihood of food sources showing up increases. Primarily, as a result, the hawks must shift their focus from hunting rabbits to acquiring them. The HHO algorithm should be required to transmit from investigation to manipulation. Mainly, as a result, a novel concept has been introduced, which also refers to the rabbit's emission energy. The has a gradual decrease before the appropriate discount rate is met, and updating is done using the following Eq. (10).
where 0 signifies the prey's initial energy in each iteration with an unfluctuating distribution in the middle of −1 and 1, and max indicates the maximum count of iterations. If | | ≥ 1 in the HHO algorithm, the Harris hawk will be using a comprehensive search strategy to find prey. If | | > 1, the indigenous search strategy is used to find prey.
Hunting Prey: During the algorithm exploitation process, the hawks will chase and eventually kill the rabbit. Real-life sequences are extremely unstable and diverse. As a result, four strategies wanted to introduce to mimic the absolute reality as much as possible, based on the different spurt modes of prey and the Harris hawk's associated jumping behaviours. It imitates a homogenous distribution in the middle of 0 and 1 during the process recycled to designate a rabbit's likelihood of successfully escaping before a surprise pounce. If r ≥ 0.5 in the HHO, it implies that the prey is unsuccessful in running; if r < 0.5, this one clearly shows that the prey will effectively spurt.
Consequently, the energy is an indicator of the predator's preferred heavy bombardment style. In HHO, if | | ≥ 0.5, the soft lay siege to is used; if | | < 0.5, the brutal lay siege is used. The specifics of each of the four actions are as follows.
Action-1: Once ≥ 0.5 and | | ≥ 0.5, indicates the soft besiege that goes into the end product. The rabbit has a lot of energy to run away; therefore, the Harris hawk will dissipate this one and encircle it for a surprise pounce. The following formula is used to explain the preceding process clearly, as in equation 11: where = (1 − 5 ) signifies the rabbit's jumping strength and 5 signifies an even distribution in the price bracket range between 0 and 1.
Action-2: While < 0.5 and | | < 0.5, indicates the Hard besiege that contrivance triggered due to the rabbit's reduced spurt energy. During that point, the prey is perceived as exhausted. The mathematical formalism is summarized as follows in equation 12: Action-3: While < 0.5 and | | ≥ 0.5, then the rabbit has more than sufficient energy to eliminate the hawk's stalking successfully. Because the prey evaded this circumstance, a Soft besiege with progressive rapid dives approach is used. Initially, the hawk resolves using Eq (13) to anticipate where they will arrive next.
The hawk will therefore take this type of tumble if is in a more competitive position-otherwise, the levy flight concept is used for this subsection. Levy flight (LF) is used in the HHO to mimic the fast, abrupt, and abnormal hawks' movement throughout hunting. With LF, the position updating formulation is as follows in equation 14: Where is a one-dimensional random list, and 1 × is the number of decision factors. The LF be mathematically signified as Eq (15).
Where and are random numbers between 0 and 1, and , is a consistent equal to 1.5. signifies the gamma distribution. In conclusion, the Harris hawk locus update management can be well-defined at this stage by the following rules in equation 16: Where is the fitness value function for the optimal selection of hyperparameters of DQN with problem minimization.
Action-4: As soon as < 0.5 and | | < 0.5, the last methodology, Hard besiege with progressive rapid dives, is used. Because the rabbit is fatigued at this point, hawks could even easily capture and try to murder it. To avoid losing anything, the hawks will reduce the average distance between themselves and the prey, i.e., contracted the invasion. Eq.(17) is used to model the behavior as mentioned above.
The is used in the HHO to mimic the rabbit's fragile carnal strength, mainly during the spurt, as calculated by Eq. (18) for each Harris hawk per iteration.
serves as a link between investigation and manipulation. If | | ≥ 1 is present, the Harris hawk will be using a comprehensive search strategy to find prey. If | | > 1 is more significant than one, the local search strategy is used to locate the prey. Besides that, if | | < 0.5, Action-2 procedures are enforced; if 1 ≥ | | ≥ 0.5, Action-1 strategies will be chosen by the hawks. | | can never really be more significant than one when the count of iterations tries to reach half of the maximum count of iterations that refers that the population is prone to reduction into local optima for roundabout multi-peak, high-dimensional complicated situations.
Only 1 's final value is more important than zero. According to Eq.(19), the prey might have enough resources to spurt even at the end of the iteration to boost the algorithm's efficiency. In the second half of the iteration, this | | cannot be substantially higher than one.
where 0 is indicates a random number within the range (−1,1); Sine and Cos signifies the sine and cosine functions, correspondingly; The total of iterations shown by , and max is represented as the maximum extent of iterations. The proposed HHO flow diagram is shown in Fig.5. Algorithm 1 provides pseudocode for optimizing hyperparameters using HHO.
The recall is expressed as a proportion of the total population of correctly labelled ovarian cyst image graphs with positive values; it is also known as the actual positive rate. The following is the approximation equation (21): Precision in ovarian cyst classification is defined as the percentage of expected positive values that are precise. The following is the approximation equation (22): As it provides the harmonic mean between precision and recall, F-measure is an enriching experience of classifier effectiveness for ovarian cyst class label generation. It is calculated using the following equation (23).  When comparing the precision of in-progress ANN, CNN and Alexnet have lower precision rates of 92%, 94%, and 95%, respectively, than the proposed HHO-DQN. The proposed method takes advantage of the HHO's faster convergence. As a consequence, the HHO-DQN is an excellent approach for ovarian cyst image classification. Table 1 shows the numerical values of the precision performance comparison.  According to the results, the proposed gains the highest recall, with the parameter optimization technique through HHO and feature extraction using CNN methods. Table 2 shows the numerical values of the f-measure performance comparison.  Compared to the proposed systems, ANN, CNN, and Alexnet, the proposed work has a lower recall rate of 92%, 94%, and 95%, indicating that it can have better grading outcomes than the existing system. Most pertinently, using an HHO and DQN-based classifier in this study tackled the issue of poor convergence and significantly decreased computing requirements for image classification, thereby improving recall value. Table 3 shows the numerical values of the recall performance comparison.   Table 4. Table 4 shows the numerical values of the accuracy performance comparison.

IoU comparison
As shown in Fig.10, the ANN classifier based on spectral indexes has the lowest IoU (0.29), indicating the least efficient strategy. With the same preparation and testing as the proposed HHO-DQN, CNN produces 0.42 IoU; however, Alexnet produces 0.52 IoU, outperforming CNN and the ANN model for ovarian cyst classification.

Fig.10. Result of IoU
However, for the ANN, CNN, and Alexnet methods, identifying or finding an acceptable threshold on class scores for classification with the specified amount of validation of ovarian cyst image samples is challenging. With 0.65 IoU, the proposed HHO-DQN outperformed ANN, CNN, and Alexnet. The proposed method highlights the importance of an active learning framework with RL in improving the accuracy of ovarian cyst classification models. Furthermore, the proposed HHO-DQN model demonstrates that incorporating HHO can provide a richer network signification, allowing for improved classification results. Table 5 shows the numerical values of the IoU performance comparison.

CONCLUSION AND FUTURE WORK
This study proposed a novel framework for automatically learning the data selection criterion for ovarian cyst image classification using a reinforcement learning algorithm. The active learning process is a sequential decision-making solution that the reinforcement learning framework will implement. Except for the previous methods, the DQN method learns the objective data function instantly rather than defining one explicitly-furthermore, HHO is used for optimizing the DQN's hyperparameters. The HHO method is achieving a better optimal solution and convergence. The proposed mechanisms automatically designed the DQN architecture for ovarian cyst classification by searching for the right set of hyperparameters.
As the experimental results show, the suggested way out has a strong case for using a CNN-based feature extractor to classify ultrasound images. While visually inspecting ultrasound images, the technology enables the radiologist to decide about the cyst's appearance. Compared with the ANN, CNN, and AlexNet models, the performance of the proposed model is better in terms of precision, f-measure, recall, accuracy, and IoU.