In Japan, approximately 400 medical-device recalls and more than 10,000 malfunctions are reported each year, leading to 100–200 actual device recalls. Using text mining, we analyzed the medical-device malfunction reports in the Ministry of Health, Labour and Welfare between 2008 and 2023. We targeted 4,529 cases on generators of cardiac implantable electronic devices, including 363 recalled cases. After mining the contents of problem status and health-damage reports, we attempted to estimate cases resulting in recalls using Bidirectional Encoder Representations from Transformers (BERT). For this purpose, we adopted tohoku-BERT, a pre-training model based on Japanese Wikipedia data, UTH-BERT, a pre-training model based on medical records, and JMedRoBERTa, a pre-training model based on medical research papers. We operated a classifier with fine tuning on a dataset annotated with medical-device malfunction reports. The UTH-BERT achieved a recall rate and F2-score of 0.931 and 0.655, respectively, on undersampled data.
Research Article
Novel Estimation of Medical-Device Recalls from Malfunction Reports using Bidirectional Encoder Representations from Transformers
https://doi.org/10.21203/rs.3.rs-4911451/v1
This work is licensed under a CC BY 4.0 License
You are reading this latest preprint version
BERT
Medical device recalls
malfunction report
Machine learning
In Japan, approximately 400 medical devices (including refurbishments) are recalled each year [1]. In addition, manufacturers and distributors in Japan report more than 10,000 medical device malfunctions each year to the Ministry of Health, Labour and Welfare (MHLW) [2], among which 100–200 are actually recalled. In Japan, medical-device recalls are classified into three categories with different risks. A Class 1 recall is made when use of the product may cause serious health damage or death. A Class 2 recall is made when use of the product will likely cause temporary or medically curable health damage. In a Class 3 recall, use of the product is very unlikely to cause moderate or serious health damage. A recall of any kind will prevent use of the medical device in clinical settings, directly or indirectly affecting medical practice. Cardiac implantable electrical devices (CIEDs) such as pacemakers require additional medical procedures such as removal or replacement of the device and modification of the program, substantially increasing the burden on patients [3]. Recalls also impose time and cost burdens on manufacturers and distributors. For example, the major European electrical-equipment manufacturer Philips reportedly resigned its chief executive officer and cut 10,000 jobs worldwide in 2023 due to the mounting costs of recalling sleep-care equipment [4]. The economic loss depends on the number of devices sold, the size of the market, and the risks posed. Meanwhile, large-scale renovations can reduce consumers’ trust in companies and products.
With the exception of recall orders issued by countries or prefectures, medical-device recalls are voluntarily made by manufacturers after receiving reports of malfunctions or health damages. Some recalls, such as poor sterilization in a particular lot, are discovered by manufacturers. Others are triggered by malfunctions or health damages during use in clinical settings. In all cases, the malfunction information is reported to the Pharmaceuticals and Medical Devices Agency (PMDA) and is published as a medical-device malfunction report at six-monthly intervals by the MHLW. During the process leading to recall, the details and causes of the defects are sometimes unknown but are commonly hinted. Even when the cause is unknown, possible or actual health damages are evidenced. Therefore, by identifying the trends of these, we could comprehend the danger and immediately initiate the recall.
Time-series data can be analyzed using Bidirectional Encoder Representations from Transformers (BERT), a Transformer-based deep learning model used in translation, text summarization, and applications such as ChatGPT developed by Google in 2018 [5]. When a large amount of data is available for learning, BERT can flexibly respond to various tasks. BERT is already widely used for extracting medical information and compiling diagnosis reports and nursing records in the medical field [6–10].
Here, we aim to construct a BERT-based machine learning model that analyzes medical-device malfunction reports in Japan over the past 15 years and thereby predicts CIED recalls.
Study Target
A total of 154,602 medical-device malfunctions reported from April 1, 2008 to March 31, 2023 were collected from the MHLW website. Each datum includes the general name and product name of the malfunctioning device, the name of the selling company, the statuses of the malfunction and health damage, the location of the malfunction, the total number of cases, and the response status. We extracted reports related to CIEDs, including pacemakers and implantable cardioverter defibrillators. After extracting pulse generators with non-blank response statuses among the CIEDs that malfunctioned in Japan, 4,529 reports were accepted for the study. All texts were composed in Japanese and the Japanese sentences were analyzed without further processing but are translated to English in the present paper.
Text mining analysis
Sentences describing the malfunction and health-damage statuses in the collected reports were analyzed using KH Coder [11], a free text-management software. First, the written text was decomposed into its smallest units through a morphological analysis. However, the morphological analysis extracted only the common words; for example, “re-operation” was divided into “re“ and “operation.” To preferentially extract the medical terms, words divided by the first-stage morphological analysis were listed as compound words if they were meaningful when combined. Such words were forcibly extracted during the second stage of morphological analysis. The extracted list and forcibly extracted terms were examined by a medical engineering researcher with a Japanese clinical engineering license.
The analysis was conducted under four conditions: (1) malfunction status, (2) health-damage status, (3) combined malfunction and health-damage status, and (4) “recalled” response status under the condition of (3). We first tallied the occurrence frequencies of words under conditions (1)–(4). To investigate the unique words in the examples leading to recalls, we selected the top 40 words extracted under condition (4) in descending order of their occurrence frequency in the sentences at the time of recall, and calculated the total number, malfunction status, and health-damage status.
BERT-based estimation of recall cases
Among the 4,529 malfunction reports, those with a “recalled” response status (363 reports) were placed in the recalled group and all others (information provision, suspension request, under investigation, and others; 4166 reports) were placed in the not group. The presence or absence of a recall was estimated using three BERT-based classification models pre-trained on large-scale data: tohoku-BERT published by Tohoku University, which is pre-trained on the Japanese version of Wikipedia [12], UTH-BERT published by the University of Tokyo, which is trained on medical records [13], and JMedRoBERTa published by the University of Tokyo, which is trained on Japanese medical research papers [14]. Data input to BERT require a morphological analysis in advance to convert the text into numerical data called tokens. In this study, some compound words were incorrectly extracted by the morphological analysis of each model. Referring to the previous results of the text-mined morphological analysis, we added these compound words separately to each BERT model. Each model with the added compound words was then trained specifically on medical-device malfunction information through tokenization and fine tuning.
The explanatory variables used for learning were set under three conditions: malfunction situations only, health-damage status only, and combined malfunction and health-damage status. To create a dataset, the ratio of training data to evaluation data in each category was set at 4:1. Because the originally targeted dataset was unbalanced, with fewer than 10% of the cases being recalled, we performed 1:1 oversampling (363: 363) and 1:1 undersampling (4166: 4166) learning in each category and compared the estimation accuracies with those of learning on the original dataset. The original and oversampled datasets were trained with a batch size of 256 over 20 epochs and cross-validated 10 times. The undersampled dataset was trained with a batch size of 256 over 200 epochs and cross-validated 10 times. The results of each language model on all datasets were fine tuned.
The task of the models was binary classification (recalled or not recalled) of the objective variable (response status). The models were evaluated in terms of accuracy (ACC), precision, recall, and weighted F-score. The ACC measures the accuracy of predicting all data. The Precision index determines the percentage of samples assigned to the recalled group that are actually in the recalled group. The Recall index determines the percentage of the recalled group that is predicted with no misses. The F-score expresses the harmonic mean of Precision and Recall. To extract the examples resulting in recall, we focus on the Recall index and hence compute the weighted Fβ-score with β = 2 (F2-score) rather than the F-score. The ACC, Precision, Recall, and Fβ-score are respectively calculated as follows:
In these expressions, TP and TN are the numbers of samples correctly assigned to the recalled group and not group, respectively, FP is the number of samples in the not group that are incorrectly assigned to the recalled group, and FN is the number of samples in the recalled group that are incorrectly assigned to the not group. The predicted number of samples in the not group is shown.
Learning was performed in a Windows 11 Pro environment with an Intel Core i9-13900KF CPU, 128 GB of memory, a 1-TB SSD, and a NVIDIA GeForce RTX 4090 (24 GB) GPU. The programming language was Python 3.8 and the deep learning library was PyTorch-Transformers.
Occurrence frequency of words
Table 1 lists the top 20 word occurrences in each status category extracted through text mining. Under malfunction status, the most frequent texts were “suspicion” and “battery” (each with more than 1,000 occurrences), followed by “early consumption,” “unknown,” and “infection.” Under health-damage status, the most frequent text was “re-operation” (>1,900 cases) followed by “infection disease,” “excision/replacement,” and “suspicion.” Under combined malfunction and health-damage status, the words “re-operation,” “suspicion,” and “battery” appeared more than 1,000 times, followed by “early consumption” and “infection.” Among the recalled items having malfunction and health-damage status, the most frequently mentioned words were “battery,” “re-operation,” and “suspicion.”
Table 1 Occurrence frequencies of words in each status category, extracted by the morphological analysis
Table 2 shows the words extracted from cases leading to recalls. Eleven words appeared in all cases leading to recalls, each appearing only a few times. Words such as “charging time,” “pacing function” and “communication failure” were included not only in more than 80% of recalled cases, but also in many cases of malfunction status. Among the words appearing at a rate of 30% in the recalled cases, “failure,” “early,” and “wastage” appeared with high frequency under malfunction status. Meanwhile, words such as “discomfort,” “pleural effusion,” and “atrial fibrillation” appeared under health-damage status but were absent under malfunction status.
Table 2 Word occurrence rates in the recalled cases
Estimation results of recall cases
Table 3 shows the estimation results of using the malfunction status alone as the explanatory variable. All models obtained ACC and Precision scores above 0.9 and 0.7, respectively, on the original dataset, but their Recall scores were approximately 0.5, meaning that they detected only approximately half of the recalled group. Under these conditions, the model trained with oversampled data using tohoku-BERT gave the best performance, with an ACC of 0.426, a Precision of 0.109, an F2-score of 0.361, and a Recall of 0.857.
Table 3 Estimation results of malfunction status
Table 4 shows the estimation results of using health-damage status alone as the explanatory variable. On the original data, all models yielded high ACC scores (> 0.9) but low Recall scores (<0.15). Undersampling and oversampling the data improved the Recall but did not sufficiently raise the F2-score of all models.
Table 4 Estimation results of health-damage status
Table 5 shows the estimation results of using the malfunction status and health-damage status as explanatory variables. On the original data, the ACC scores of all models were 0.95 or higher but the Recall scores were in the 0.5 range, meaning that only approximately half of the recalled group could be estimated. The tohoku-BERT model trained on oversampled data achieved a high Recall (0.934) but its F2-score was low (0.952). On the contrary, the JMedRoBERTa model trained on undersampled data achieved both a high Recall (0.928) and a high F2-score (0.639). Furthermore, when learning with UTH-BERT on undersampled data, Recall was 0.931 and the F2-score was 0.655, confirming a relatively well-balanced model.
Table 5 Estimation results of combined malfunction and health-damage status
Various reports on medical-device recalls in Japan have been published, including studies analyzing the nature of recalls and the time period of their occurrence [15, 16], studies analyzing the causes of recalls [17], and studies focusing on medical materials and the impact of recalls on the medical field [18]. Other studies have investigated bugs targeting medical-device programs and software [19]. Here, we constructed a model that predicts cases leading to recalls among the 10,000 reports of medical-device malfunctions published each year. Unlike a previous study based on decision trees [20], it uses only the medical-device malfunction reports over a single year, but covering all medical devices (Classes I–IV). Such a model must be closely scrutinized for sufficient estimation. Among approximately 154,600 malfunction reports collected over the past 15 years, only approximately 0.01% (~ 2,000 cases) resulted in a recall. Therefore, we considered that a model that combines these would yield poor estimation accuracy and incur a high computational cost.
In Japan, approximately 3–5 medical devices recalled per year are in Class I. CIEDs (which are Class IV devices) are most commonly recalled and are frequently recalled because of high risk. Therefore, we considered that recalls will notably affect medical practices and companies. The CIEDs (specifically, pulse generators) targeted in our study are often combined with accessories such as leads. Over 10,000 reports of malfunctions, including those related to accessories, have been filed, but only around 500 cases of CIEDs, including accessories, have been recalled, implying a large degree of data imbalance and poor estimation accuracy. To improve the estimation accuracy, we limited our study to the pulse generators of CIEDs, which have a high recall rate.
A frequency analysis of the extracted words suggested that “problems related to premature battery consumption,” including “problems related to early battery consumption,” “infections not originating from the product,” and “pacing failure,” including suspicions, will likely be reported as malfunctions. Meanwhile, specific health damages include “re-operation or fear of re-operation,” “infection or suspected infection,” and “removal/replacement of the generators.” This information presents in both recalled and non-recalled cases. As shown in Table 1, very few words are unique to the recalled group. Meanwhile, over 90% of the problems caused by (for example) “charging time” or “pacing function” have resulted in recalls. Such words are likely to become characteristic words representing recalls during BERT estimations. However, only around 20 words in the recalled group have occurrence frequencies above 50%. The recalled group in the dataset of this study is small (363) and correct identification of its words is deemed crucial for improving the estimation accuracy.
Here, we aimed to construct a model that can predict all cases leading to recall. To focus on the Recall measure, we replaced the usual F-score, the harmonic mean of Precision and Recall, with the F2-score. An Fβ-score of 0.5 or higher is considered appropriate and a value over 0.6 is considered high. Our dataset was highly unbalanced, with the recalled group constituting less than 10% of all samples. Therefore, learning based on this dataset will be biased toward the data with a large proportion (the not group), improving the accuracy of judging the not-group samples. Accordingly, the ACC scores of all BERT-based models were high (> 90%), regardless of explanatory variable. However, the Recall scores indicating the recall detection accuracies were approximately 60% for malfunction status and approximately 10% for health-damage status. Although some models achieved high ACC and Recall scores for health-damage status, their F2-scores were all below 0.5, suggesting that correct learning is not possible on the original dataset. After under/oversampling the data, the ACC and Precision scores decreased but the Recall exceeded 70% under all conditions. The model trained by tohoku-BERT on oversampled data achieved a particularly high Recall for malfunction status in the recalled group (0.934; 24 errors out of 363 data). However, the F2-score of this model was low (0.392). The most consistent model was UTH-BERT trained on undersampled data, with a Recall of 0.931 and an F2-score of 0.655 for malfunction and health-damage status.
One limitation of this study is unification of the notation of the written content. A glossary of medical- device malfunction terms has been compiled by the International Medical Device Regulatory Authority Forum and published by the PMDA [21]. The descriptions of malfunction status are in principle based on this glossary. In contrast, the definitions of terms related to health damages, such as “unwell,” “feeling unwell',” “chest tightness,” and “chest pain” are ambiguous and depend on the authors of the reports. In this study, tokenization and learning were performed in a mixed state; that is, as separate words. Therefore, in future work, these spelling variants must be carefully examined and (in some cases) unified before learning. In addition, these examinations were overseen by a single researcher (a clinical engineer). More accurate forced extractions for synonym unification are expected with assistance from engineers and recalls personnel in manufacturing and sales companies. However, if a word does not characteristically appear in the cases leading to recalls, it will probably negligibly affect the estimation accuracy. For example, as tohoku-BERT did not correctly extract the word “re-operation” during the morphological analysis, the word was registered in an additional dictionary, although its proportion in the recalled group was only 1/9th that in the not group. Therefore, the estimation accuracy is unlikely to noticeably change if “re-operation” is divided into “re“ and “operation.” In contrast, the estimation accuracy can be improved by ensuring that words such as “pacing function” and “charging time,” which frequently appear in recollection cases, or words that are unique to recalled cases, are registered in the dictionary. Therefore, to improve the estimation accuracy, the characteristic terms in recalled cases must be extracted in the correct form for learning.
Expanding the range of target devices is another avenue for future research. As malfunctions and health damages depend on the target device, we will optimize the morphological analyses of different devices, extract words unique to each recovered case, and aim to further improve the estimation accuracy.
We constructed a natural language processing model that analyzes the malfunction and health-damage statuses of implantable devices (pacemakers and cardioverter defibrillators) from the past 15 years of medical-device malfunction information. After analysis, the model estimates the recalls. The UTH-BERT-based model estimated the recalled devices with an accuracy of 0.931 and a weighted F (β = 2) of 0.655 on the undersampled dataset of malfunction and health-damage statuses.
Author contributions
KI conceived and designed the study. KI encoded and input the data into BERT. KI developed the study protocol, statistical analyses. KI wrote the manuscript.
Funding
This work was supported by research grant of Medical Device Strategy Institute Japan
Data Availability
Data is available upon reasonable request.
Ethical approval
This was not a study of human subjects, but an analysis of the results of an educational examination routinely conducted. Therefore, neither receiving approval from the institutional review board nor obtaining informed consent was required.
Competing interests
The author declare that there is no conflflict of interest regarding the publication of this paper.
- Medical device recall information. Ministry of Healthc, Labour and Welfare. https://www.mhlw.go.jp/stf/seisakunitsuite/bunya/kenkou_iryou/iyakuhin/kaisyu/
- Information regarding case reports of suspected malfunctions. Pharmaceuticals and Medical Devices Agency. https://www.info.pmda.go.jp/asearch/jsp/menu_fuguai_base.jsp
- Kapa S, Hyberger L, Rea RF et al (2007) Complication Risk with Pulse Generator Change: Implications When Reacting to a Device Advisory or Recall. PACE, 30:730–733. https://doi.org/10.1111/j.1540-8159.2007.00742.x
- Dutch electronics giant Philips to cut 6,000 jobs worldwide. AP NEWS. (2023) https://apnews.com/article/philips-electronics-layoffs-faf061f38fea2cf83cb7dba803c5d670
- Devlin J, Chang MW, Lee K et al (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proc. of NAACL-HLT2019, 4171–4186. https://doi.org/10.18653/v1/N19-1423
- Kita K, Uemura K, Takao M et al (2023) Use of artificial intelligence to identify data elements for The Japanese Orthopaedic Association National Registry from operative records. J Orthop Sci., 28:1392–1399. https://doi.org/10.1016/j.jos.2022.09.003
- Denecke K, May R, Rivera RO (2024) Transformer Models in Healthcare: A Survey and Thematic Analysis of Potentials, Shortcomings and Risks. J Med Syst., 48:23. https://doi.org/10.1007/s10916-024-02043-5
- Olthof AW, van Ooijen PMA, Cornelissen LJ (2021) Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance. J Med Syst., 45:91. https://doi.org/10.1007/s10916-021-01761-4
- Sakamoto K, Kishimoto K, Yamamoto T et al (2022) Attempt of Automatic Audit for Nursing Records Using BERT: Self-removal of Tubes in Medical Record Entries. Trans. of Jpn. Soc. for Med. Biol. Eng., 60:95–101. https://doi.org/10.11239/jsmbe.60.95
- Shibata D, Kawazoe Y, Shimamoto K et al (2020) Pain Expressions Extraction with Pre-training BERT Based on Clinical Note. Jpn J. Med. Inf., 40:73–82. https://doi.org/10.14948/jami.40.73
- Higuchi K (2016) A Two-Step Approach to Quantitative Content Analysis: KH Coder Tutorial Using Anne of Green Gables (Part 1). Ritsumeikan Soc. Sci. Rev., 52:77–91.
- Suzuki, M. cl-tohoku/bert-japanese: BERT models for Japanese text. https://github.com/cl-tohoku/bert-japanese (accessed: 2-Jul-2024).
- Kawazoe Y, Shibata D, Shinohara E et al (2021) A clinical specific BERT developed using a huge Japanese clinical text corpus. PLoS ONE, 16: e0259763. https://doi.org/10.1371/journal.pone.0259763
- Sugimoto K, Iki T, Chiba Y et al (2023) JMedRoBERTa: a Japanese Pre-trained Language Model on Academic Articles. Proc. of the 29th Annual Meeting of the Association for Natural Language Processing, 707–712.
- Mori K, Nakano S, Kasanuki H et al (2014) High risk recalls of medical devices in Japan: A comparison in recall rate, recall lag and decision of recall classification with the United States. Regulatory Science of Medical Products, 4:103–110.
- Ishida K, Hirose M, Tsuruta H (2014) Analysis of quinquennial medical device recalls in Japan. JJMI, 84:387–395. https://doi.org/10.4286/jjmi.84.387
- Nakayama M, Tanaka S, Hamada S et al (2018) Recalls and Premarket Review Systems for High-Risk Medical Devices in Japan. Ther Innov Regul Sci., 54:775–780. https://doi.org/10.1177/2168479018812812
- Sakamoto M, Okubo T (2006) Survey Report of Recall Due to Sterilization Failure in Health Care Facilities in Japan. JJMI, 76:21–30. https://doi.org/10.4286/ikakikaigaku.76.1_21
- Hiraga H, Uekusa S (2023) Investigation of Medical Device Program Defects Affecting Pharmacotherapy. J Drug Inform., 25:758–761. https://doi.org/10.11256/jjdi.25.91.
- Ono H, Yokoi H, Nakazono M (2016) Estimation of the events leading to field safety corrective actions(FSCA) from adverse event reports of medical device. Jpn J Med. Inf., Suppl.:758–761.
- IMDRF Glossary of Medical Devices in Japan. Pharmaceuticals and Medical Devices Agency. https://www.pmda.go.jp/files/000265827.pdf
No competing interests reported.