DOI: https://doi.org/10.21203/rs.3.rs-487369/v1
Background Invent Pill classification system that can detect and classify into one tablet unit by deep-learning and Pill filming system that generate comprehensive and multi-dimensional data for learning.
Methods Pill filming system and Pill classification system, they have two chapters consisted of structure design, model introduction, and controller design. Pill classification system's structure categorization is Input box, Linear conveyor, and Output box. In Data preprocessing, a similarity map is obtained with Structure Similarity Index Measure(SSIM). And RetinaNet is used as a pill classification learning model. Mean Accuracy Precision (mAP) is 0.9842, and we take experiment about measuring the number and accuracy of the classified pills for each experimenter's classification time.
Conclusion Pill filming system and Pill classification system are expected to reduce labour losses for simple tasks. It helps medical personnel focus on significant and urgent tasks. And It can contribute to experiments about that deep-learning control the mechanical device.
The hospital's drug department provides medicine to the patient according to the doctor's prescription. However, depending on the patient's condition, the prescription changes frequently occur, resulting in prepared but not administered drugs[1–3]. These drugs are recovered and require reclassification[4]. The reclassification process is currently carried out through manual tasks directly checked by two or more pharmacists or nurses[5]. This process results in hours of labour loss for simple work[6].
Recently, COVID-19 broke out and increased the use of drugs in hospitals and also the number of recovered drugs[7]. It results in not satisfying the demand for reclassifying and spotlights the effectiveness of medical personnel's labour[8].
There are devices named pill classification in the market, but it can only classify boxes or pack units. Co. Emtros patented about Classification apparatus of recovery pills in 21014. However, it was not based on deep-learning but can recognize based on class with shape: circle, ellipse, and capsule and with color: ivory, red, sky blue, orange, black, white, and blue[9]. So it has problem with classifying pills, which have similar look. Therefore there is never system that can detect and classify just one tablet by deep-learning.
Therefore, a deep learning-based pill classification system is needed to classify pills automatically to reduce labour losses for simple tasks. It helps medical personnel focus on significant and urgent tasks.
Deep learning technology increases the effect of more data available for learning[10]. Malaya university had experiment about pill identification using deep convolutional network. It used 10 ~ 25 pictures for each pill, but it is not enough amount of data for learning and taking pictures of pills by hand is cumbersom[2]. However, securing large amounts of data sets is not easy[11]. As a result, many attempts operate to augment data through simple data expansion techniques such as rotation, reversal, and filtering[12–13]. However, these techniques are limited in scalability because they use only the data they already have.
Therefore, to classify high-accuracy pills, Pill filming system is proposed to generate comprehensive and multi-dimensional data through the operation of motor-controlled-based seesaw structures.
This paper presents the Pill filming system and Pill classification system in two stages, and they consist of struct design, model introduction, and controller design.
Figure 1 is the prediction result of the pill image captured by the Pill classification system’s camera. Table 1 shows the results of testing the verification data for the model selected in this paper. To check the performance of the algorithm, average precision (AP) per class is calculated. Mean average precision (mAP) obtained by dividing the sum of all AP by the number of classes is 0.9842.
|
Class |
Accuracy Precision (AP) |
|---|---|
|
0 |
1.0 |
|
1 |
1.0 |
|
2 |
0.98 |
|
3 |
0.9412 |
|
4 |
1.0 |
|
Mean Accuracy Precision (mAP) |
0.9842 |
To verify the performance of the Pill classification system, we experimented with measuring the number and accuracy of the classified pills for each experimenter's classification time. The experiment was conducted on 100 of the five pills used in this paper. In Fig. 2, when each graph ends, the speed was fast in the order of pharmacist, nurse, Pill classification system, and non-medical. However, the slope of the graph means that the device was the most constant. That is, the variation of concentration was slight. Also, the accuracy of pill classification by experimenter is shown in Table 2. The accuracy of classification by the pill classification system was the highest at 0.99. Therefore, it is more efficient to classify pills with Pill classification system when classifying large amounts of pills.
|
experimenter |
Pharmacist |
Nurse |
Non-medical |
Pill classification system |
|---|---|---|---|---|
|
Accuracy |
0.96 |
0.95 |
0.89 |
0.99 |
Pills recovered to the pharmaceutical department require much labour to reclassify. COVID-19 broke out and spotlighted the effectiveness of medical personnel’s labour. There are devices named pill classification in the market, but it can only classify boxes or pack units.
To address this problem, we suggest Pill classification system that can detect and classify just one tablet. Deep learning-based automatic pill classifier can solve this, but it needs image data taken from various angles. Therefore, in this paper, we also designed Pill filming system that can generate multi-angle and multi-dimensional image data to be learned by an automatic pill classification system. Figure 3 shows fabrication of the Pill filming system and Fig. 4 shows fabrication of the Pill classification system we produced.
We take two experiments, one is about the number of classified pills per time(second), and the other is measuring accuracy. This system has the speed for running equally without acceleration and makes user calculating the time for a specific amount of pill reclassifying. It has the highest accuracy at 0.99 in the experiment with pharmacist, nurse, and non-medical. But it is just that one person of job-pool taken and it has potential difference between personal. It can be supplemented by more experiments.
The Pill filming system and Pill classification system are expected to reduce labour losses for simple tasks. It helps medical personnel focus on significant and urgent tasks.
And It can contribute to experiments about that deep-learning control the mechanical device.
Fig. 5 shows that DC motor, Motor cap, Columns place on Base plate, and Crank connects with Seesaw and Camera guides support Camera holder.
The device operates with converting the DC motor's rotary motion to linear motion through the Crank, and it makes Seesaw running. When the Seesaw operates, the Columns next to each side of the Seesaw fix the Seesaw's motion axis.
From fig.6, there are hills inside the Seesaw, and when the motor activates, the hill randomly changes the way the pill looks at the camera. When the pill places under the camera, the camera shoots the pill.
The Pill filming system's overall size is 195 mm × 189 mm × 206 mm, and it weighs 0.52 kg, not including the pill. This system was manufactured with PLA using a 3D printer for a lightweight and easy manufacturing process.
Fig.7 and Fig.8 show that the Pill filming system consists of power supply, board, motor controller and motor[14]. The motor controller consists of motor drive L293D, which can control the motor and the DC motor provides a major force to the device. The DC motor's speed is 15000 RPM, and the Gear Reduction Ratio is 150:1, and consequently, the number of rotations is 100 RPM. The motor speed was controlled by setting the Dutycycle to 13%, thus obtaining approximately 14 images per minute.
User interface was implemented using QT designer, as shown in fig.5. Once you click on the UI button, 300 images like fig.10 place in each folder.
The Pill classification system consists of Input box, Linear conveyor, and Output box with different heights in Fig. 11, and they operate in regular order. User through unclassified pills into the Input box, it put off one pill on the Linear conveyor. Linear conveyor puts the pill under camera for classification and takes the pill to Output box. The Output box turns as fitting the pill, and then the Output box gets the pill inside case.
Fig. 12 shows each part’s name in Pill classification system. In the Input box, Pillbox gets lots of pills inside, and Servo motor under Pillbox hit Slider vertically. Slider has six Threshold which is different from each other, so when Slider gets hitting, pills slide according to Slider’s way, and one pill throw out itself onto the Linear conveyor (A). Detector SW has position Slider’s terminal.
When the pill attaches to Linear conveyor (A), DC motor at the Linear conveyor's side runs to move the pill under Camera holder. After classification about what pill is, the Output box turns as fitting to that pill’s Pill case. Then DC motor makes the pill dropping at Linear conveyor (B), and the Output box turns to its original position.
Step motor which is fixed in Baseplate control Turnplate to turning. The Turnplate has six Pill case and each Pill case match for pills.
Baseplate fixes every parts’ position and has a control board inside. Pillar (A) suspend Pillbox, and Pillar (B) suspend the Camera holder.
All part excepting Baseplate was produced with material PLA using a 3D printer for the easy manufacturing process and Baseplate is made of aluminum. The Pill classification system's overall size is 680 mm × 220 mm × 302 mm, and it weighs 9.08kg, not including the pill.
As shown in Fig. 13, a total of 5 pills are used from Gachon University Gil Hospital in the data set. After obtaining 300 images of multi-dimensional pills for each type by operating a manufactured pill filming system, 240 are used for training data, and 60 are used for validation data.
The image annotation process is performed to extract the pill that is the region of interest from the obtained pill image. After adding pill images for each type, average them to make a reference (background image). A similarity map is obtained with Structure Similarity Index Measure(SSIM), a structural similarity index that measures image quality. It is a structure that estimates the region with the lowest value by calculating the local Norm value corresponding to 170 pixel x 170 pixel in consideration of the pill size.
To speed up the operation, wavelet transform is on the similarity map to remove horizontal, vertical, and diagonal components, and then extrapolation to make the original size. Also, when calculating the local norm, the process of skipping the pixels exceeding the reference value is performed.
In this paper, RetinaNet is used as a pill classification learning model, and it is selected as a model capable of simultaneous detection and classification in consideration of the differences in the environment of pill filming system and pill classification system.
RetinaNet is an integrated network consisting of a ResNet-based Backbone, and two subnetworks, a class subnet and a box subnet, as shown in Fig. 15 [15].
The epoch used for training is 100, the batch size is 12, and the learning rate is 0.00001.
Fig.16 is a block diagram of Pill classification system, and Fig.17 is a circuit of Pill classification system. Control board gets voltage 5v from power supply and makes system operating[14].
In Input box, the board sends signal about angle, which control servo motor for a pill dropped and gets detector switch’s input value to find out when the pill touches the detector switch.
External power supply gives voltage 12v to Linear conveyor’s motor drive, L293D, which controls DC motor. The DC motor in the Linear conveyor provides a major force to conveyor belt for running. The DC motor’s speed is 15000 RPM, and the gear reduction ratio is 150:1, and consequently, the number of rotation is 100RPM.
Output box has the same motor drive L293D, and it gets voltage 12v from the external power supply. The motor drive controls step motor, which operates 1.8° per 1 step.
Fig. 18 shows the algorithm flow diagram of the executable code for pill classification system. By controlling the servo motor from 25° to 130°, hit the structure with a constant jaw between the input box and the linear conveyor, and drop the pill from the input box one by one. The frequency was set to 50 Hz and 30 % duty-cycle speed, controlling the DC motor through PWM control, and rotating the motor once to position the pill in the camera frame at the top. After that, the camera is captured by the detector switch's input signal, followed by detection and classification. As the predicted result, the step motor is controlled based on 60° between each pill container so that the container corresponding to each pill comes to the end of the linear conveyor. By setting the frequency to 50Hz and the duty-cycle to 65%, the DC motor is rotated three times through PWM control to operate the linear conveyor. Then the pills are dropped into the appropriate pill case. After all this process is over, it goes back to its original position.
This article does not contain any studies with human participants or animals performed by any of the authors.
This article does not contain patient data.
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
This article does not have funding.
Juhui Lee designed Pill filming system and Pill classification system’s mechanical part and was a major contributor in writing the manuscript. Soyoon Kwon designed algorithm that Pill filming system and Pill classification system operate and was a major contributor in writing the manuscript. Jong Hoon Kim and Kwang Gi Kim gave advice for solving problem in research processing. All authors read and approved the final manuscript.
No potential conflict of interest relevant to this article was reported.
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2020-20170-01630) supervised by the IITP(Institute for Information & communications Technology Promotion), and the Gachon Unversity(2019-0369).
Juhui Lee and Soyoon Kwon contributed equally to this paper.