An Intelligent System for Surgical Tools Checking, using Vision and Deep Learning

doi:10.21203/rs.3.rs-3801061/v1

Download PDF

Short Report

An Intelligent System for Surgical Tools Checking, using Vision and Deep Learning

https://doi.org/10.21203/rs.3.rs-3801061/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Objective: Currently, Operating Rooms (OR) check surgical tools manually before and after every surgery and sterilisation, making the process tedious. In addition, each hospital’s OR teams have different work processes, making standardisation challenging and staff training obsolete as they move from one hospital to another. Coupled with manpower challenges in healthcare, this makes ORs significantly inefficient. We propose a standardised system which leverages on Deep Learning to increase inter-organisational effectiveness by expediting tool accounting. Photos of surgical tools were taken, making up a dataset which we trained with using TensorFlow API. We tested the detection confidence of our trained models on each surgical tool and tabulated our results.

Results: Our initial test showed many False Negative results, with True Positive results for only 3 tools. After adjusting our model training process for the second test, an improvement was seen with only 1 False Negative result produced. Our project shows potential for deep learning to be used in the future to streamline backend surgical processes, if limitations such as similarities between tools, inconsistent number of training images, and insufficient training images are worked on.

Surgical tools

Deep Learning

Image recognition

Single-Shot Detection

Regions with Convolutional Neural Networks

Confusion Matrix

Background of Research

Modern Operation Room (OR) protocols mandate medical personnel manually check for the presence of all surgical tools before an operation, after the operation, and following the transfer of the surgical tools to sterilisation areas, and that these are intact, and usable (1). While this reduces preventable errors like the use of defective surgical instruments or retained surgical instruments (2), which may result in consequences ranging from financial costs to the loss of human lives (3), this method surgical tool identification and accounting is time-consuming. Differences in the workflow of different OR teams in different hospitals further reduces inter-organisational effectiveness.

The duration of intervals between operations could be reduced with the development of a surgical tool recognition system that eliminates the need for human intervention, as multitudes of other OR-related procedures, such as automated clinical assistance, and staff assignment, have been shown to benefit from surgical workflow automation (4).

Prior studies have utilised Computer Vision and Machine Learning (5) to account for surgical tools, with varying degrees of success. Compared to traditional Computer Vision techniques, such as edge detection, corner detection, and template matching, Deep Learning achieves greater accuracy in object detection. The large amount of data on surgical tools we can gather from taking photos in a controlled lab setting allows Deep Learning models to be trained on a vast bank of data, significantly improving accuracy of detections relative to those that are hard coded into Computer Vision algorithms. Deep Learning also provides increased flexibility, especially when minor changes are made to the dataset to be trained to fine-tune the object detection model (6). This allows for greater ease of use and efficiency when it comes to improving the entire object detection system.

Consequently, this study envisions a surgical tool-checking system modelled using Deep Learning. We foresee an eventual system where, at various checkpoints of transport and checking, Deep Learning models integrated into cameras and 3D scanners simultaneously account for surgical tools and check for defects or distortion. This streamlines the entire process, reducing wait times and enabling more operations to take place, whilst ensuring the accuracy and reliability of checking surgical tools before and after surgery.

Hypothesis of research

We hypothesise that a system equipped with Deep Learning capabilities will increase the efficiency of accurate checks of surgical tools done before and after surgery, while maintaining an acceptable level of confidence in the detection model.

Obtaining Dataset

To obtain the dataset of the surgical tools needed for training and testing before training our model, we constructed our own library of surgical tool images. To ensure clarity of images and uniformity of dimensions, we had a controlled lab setting to capture photographs of the tools. Two Logitech BRIO webcams were set up on a retort stand (Fig. 1). One webcam was placed directly above the tool (90°), and the other on the horizontal axis (45°). These webcams were linked to a computer running a Python script to take photographs automatically. The tools were set on a turntable that allowed the webcam to capture images of the tools at various angles. This allowed a diverse range of images to further enhance our Deep Learning model.

We utilised the OpenCV library for the Python script, which accessed the two webcams at the same time while the turntable spun.

Preparing Dataset

To prepare the dataset to train our model, manual image labelling and classification was done. The labelImg Python library was implemented for this (Fig. 2) to label the images in the PASCAL VOC format.

Training Our Model

We utilised the SSD ResNet50 pre-trained model (7) in the TensorFlow API model garden for real-time detection and accuracy. In considering both TensorFlow CPU and TensorFlow GPU, we selected using the latter, due to significant increases in performance efficiency. Per-step time in the former at an average of ~ 55s (Fig. 3A) falls short by a large margin compared to the latter at an average of ~ 1.0s (Fig. 3B).

Upon completion of the training procedure, the trained model was utilised and fed into a detection script that combined the Python CV2 library with the Deep Learning model. This was done so through a Jupyter Notebook script that can set up the necessary libraries required to draw information from the model and run the CV2 script without any interruptions. This allowed us to successfully curate a surgical tool detection algorithm using Deep Learning, while also obtaining the relevant details on the accuracy of detections (Fig. 4).

Evaluation of Training Model

To ascertain the effectiveness of our training model, we used 5 test images of each surgical tool, with each tool being placed at different orientations and positions in the image. Additionally, we decided to use photos of the tools in pairs to see how effective our model would be in recognising multiple tools at the same time (Fig. 5).

After the first test, we modified some aspects of the training process to improve on the accuracy of our detection model. This included limiting our classes to only Dental Extracting Forceps C and the Endoscope, while using Faster-RCNN instead of SSD ResNet50 as it emphasised more on detection accuracy rather than speed (8). The model was then tested with 5 test images of each surgical tool.

Results from First Test

We classified our results based on a Confusion Matrix (9). Of the 34 images of surgical tools we tested the system with, only 3 images yielded a TP result, while 5 images generated FP results, with the rest of the images being a FN result (Fig. 6).

In analysing the 3 TP and 5 FP results, which were namely the Endoscope, Dental Extracting Forceps C, and the Metz Scissors, we came to two conclusions. Firstly, the tools which has unique physical characteristics specific to it itself allowed for successful recognition. A contrast to the background which helped to accentuate the characteristic also helped to increase detection accuracy. Secondly, tools with similar features such as the Metz Scissors and Crile Hemostat will be classified as the tool with the simplest features, and in this case, the Metz Scissors is considered the simplest as it only contains the rounded loops and sharp edges at the tip.

Results from Second Test

Out of the 16 tools, 12 yielded TN results, 2 yielded FP results, while the Endoscope had a TP result, and the Dental Extracting Forceps C had an FN result (Fig. 7).

Analysing these results confirmed our findings from the previous test, as focusing on the FP results and the FN result showed that tools with similar characteristics will be classified as the tool with more simple features. The TP result from the Endoscope confirmed that surgical tools with more unique characteristics have a higher accuracy when fed into the model testing algorithm.

From the tests, efficiency increased dramatically with the adoption of Deep Learning capabilities in the surgical tool checking process. Manual checks took us around a minute to check through a set of 16 tools, while the model took 15 seconds. Amongst those whose predictions failed, similarity between features of the surgical tools, the difference in the number of training images for each tool and lack of sufficient training images were responsible for most errors.

Similarities between Features of Surgical Tools

Amongst the surgical tools we used, a few of them had similar features, namely the two loops which function as the handles for the tools found in Allis Forceps, Artery Forceps, Crile Hemostat, Mayo Scissors, Metz Scissors, and the Weitlaner Retractor, and the thin metallic body found on the Dural Separator, Ear Hook, Gilles Skin Hook, and Scalpel Handle. These similarities in features may lead to the Deep Learning model confusing these surgical tools, as most current prevailing Deep Learning models all seek to identify common larger objects with greater differences between each other, such as buses, trees, and humans. In addition, we are unable to use algorithms like YOLOv7, an algorithm that emphasised training speed while compromising on details, to train our model due to the nature of surgical tools being so similar to one another.

Difference in Number of Training Images

Due to the nature of the Deep Learning model, the imbalance in the number of training images within the dataset would have severe negative implications on the model’s ability to differentiate between the classes of surgical tools, especially with the high complexity involving identifying similar-looking surgical tools. An imbalanced training dataset results in a biased learning model with too little training data for minority classes, resulting in poor predictive accuracy over the aforementioned classes which causes underfitting, signifying that the model was unable to capture the underlying pattern (10). A suitable method of resolving this issue would be “oversampling”, which is to extend the size of the minority class and is effective at addressing the imbalance with notable improvements to performance (11).

Lack of Sufficient Training Images

In training our model, the lack of sufficient images in our dataset could have resulted in underperformance due to the inability of the Deep Learning algorithm to gain exposure of variance in each detection class. Therefore, when testing the model with brand new images previously not seen by the algorithm, it was unable to generalise to all the possible combinations of tool images.

While our first model achieved a modest success rate of only 8.8%, with 3 out of 34 accurate predictions, we are confident that addressing the issues mentioned by expanding our dataset to include higher contrast to improve the comprehensiveness of our dataset, and standardising the number of photos taken for each surgical tool would significantly improve our model’s accuracy. On top of that, a conscientious selection process of the model used will also be necessary, as the complexity with detecting small differences between similar-looking surgical tools calls for a more comprehensive model. An optimal model can be obtained through repeated trial-and-error.

Limitations

Our results were based on the dataset which comprises photos taken of the various surgical tools. However, in working with surgical tools, we encountered some limitations. Firstly, most object detection software is focused on distinguishing everyday objects which possess many differences. Surgical tools are extremely similar in shape and are of the same colour, thus it made it difficult for the software to accurately detect the differences between two similar surgical tools. To improve on the accuracy of the detection algorithm, it is necessary to select an optimal comprehensive model that differentiates between tools by their small differences, eliminate class imbalance in the training dataset, and expand the dataset to contain images with higher contrast. Secondly, due to the need to magnify the features of each surgical tool, we had to place the camera close to the surgical tool. However, in a surgical setting, the need to put the camera close to the surgical tools would mean that nurses in the operating theatre would have to pass each surgical tool individually under the camera. This would increase the time taken for the process, as compared to our initial plan which was to take a photo of all the tools at once and recognise each tool, ultimately compromising on the efficiency of the entire surgical tool checking process. One possible adjustment to resolve this issue would be to train the model with multiple tools and annotations in one image, thus acquainting the model with an entire set of tools in a single round of detection, increasing efficiency.

True Positive

True Negative

False Positive

False Negative

SSD

Single Shot Detection

R-CNN

Regional Convolutional Neural Network

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This study was partially funded by the National University of Singapore. The National University of Singapore provided technical support for the present study.

Authors' contributions

All authors conceived and explored the idea of this research topic. MC and YP collected and organised the data. DW trained and tested the deep learning model. All authors drafted the manuscript, together with reading and approving it.

Acknowledgements

We would like to appreciate Prof. Tang Kok Zuea from the National University of Singapore for the technical support as well as the invaluable advice provided throughout the duration of the research endeavour.

WHO Patient Safety, Organization WH. WHO guidelines for safe surgery 2009: safe surgery saves lives [Internet]. World Health Organization; 2009 [cited 2023 Jul 18]. Report No.: WHO/IER/PSP/2008.08-1E. Available from: https://apps.who.int/iris/handle/10665/44185.
Weprin S, Crocerossa F, Meyer D, Maddra K, Valancy D, Osardu R, et al. Risk factors and preventive strategies for unintentionally retained surgical sharps: a systematic review. Patient Saf Surg. 2021;15(1):24.
Lt K, Jm C, Ms D, PubMed. 2000 [cited 2023 Jul 18]. To Err is Human: Building a Safer Health System. Available from: https://pubmed.ncbi.nlm.nih.gov/25077248/.
Zayas-Cabán T, Haque SN, Kemper N. Identifying Opportunities for Workflow Automation in Health Care: Lessons Learned from Other Industries. Appl Clin Inform. 2021;12(3):686–97.
Mahony NO, Campbell S, Carvalho A, Harapanahalli S, Velasco-Hernandez G, Krpalkova L et al. Deep Learning vs. Traditional Computer Vision [Internet]. Vol. 943. 2020 [cited 2023 Jul 18]. Available from: http://arxiv.org/abs/1910.13796.
Rodrigues M, Mayo M, Patros P, OctopusNet. Machine learning for intelligent management of surgical tools. Smart Health. 2022;23:100244.
GitHub [Internet]. [cited 2023 Aug 31]. models/research/object_detection at master · tensorflow/models. Available from: https://github.com/tensorflow/models/tree/master/research/object_detection.
Hui J. Object detection: speed and accuracy comparison (Faster R-CNN, R-FCN, SSD, FPN, RetinaNet and… Internet]. Medium. 2019 [cited 2023 Aug 31]. Available from: https://jonathan-hui.medium.com/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359.
Sammut C, Webb GI, editors. Encyclopedia of Machine Learning [Internet]. Boston, MA: Springer US; 2010 [cited 2023 Jul 18]. Available from: http://link.springer.com/10.1007/978-0-387-30164-8.
Zheng W, Jin M. The Effects of Class Imbalance and Training Data Size on Classifier Learning: An Empirical Study. SN Comput Sci. 2020;1(2):71.
Buda M, Maki A, Mazurowski MA. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 2018;106:249–59.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

An Intelligent System for Surgical Tools Checking, using Vision and Deep Learning

Status:

Version 1

Abstract

Figures

Introduction

Background of Research

Hypothesis of research

Methodology

Obtaining Dataset

Preparing Dataset

Training Our Model

Evaluation of Training Model

Results and Discussion

Results from First Test

Results from Second Test

Discussions

Similarities between Features of Surgical Tools

Difference in Number of Training Images

Lack of Sufficient Training Images

Limitations

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Authors' contributions

Acknowledgements

References

Additional Declarations

Status:

Version 1