Application of Deep Learning Technology for Temporal Analysis of Videofluoroscopic Swallowing Studies

doi:10.21203/rs.3.rs-2311543/v1

Download PDF

Research Article

Application of Deep Learning Technology for Temporal Analysis of Videofluoroscopic Swallowing Studies

https://doi.org/10.21203/rs.3.rs-2311543/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: For an objective and quantitative evaluation for videofluosroscopic swallowing study (VFSS), temporal parameters during swallowing are analyzed for both clinical and research purposes. However, because it is time-consuming and is complicated to analyze manually by clinicians, automated analysis using deep learning has been attempted. This study aimed to develop a model for automatic measurement of various temporal parameters of swallowing using deep learning.

Methods: Five-hundred and forty-seven VFSS video clips were included in this study. Seven temporal parameters were manually measured by two physiatrists as ground-truth data: oral phase duration, pharyngeal delay time, pharyngeal response time, pharyngeal transit time, laryngeal vestibule closure reaction time, laryngeal vestibule closure duration and upper esophageal sphincter opening duration. ResNet3D was chosen as the base model for deep learning of temporal parameters. Specifically, we proposed several architectural variants of ResNet3D to improve the estimation of temporal parameters. The performance of ResNet3D variants was compared with that of the VGG and I3D models which were used in the previous studies.

Results: The average accuracy of the proposed ResNet3D variants in measuring the temporal parameter of VFSS was from 0.864 to 0.981. F1 score ranged from 0.707 to 0.924, and average precision ranged from 0.643 to 0.865. In comparison with the VGG and I3D models, our model achieved the best results in terms of accuracy, F1 score and average precision values.

Conclusion: We proposed deep learning models for automatically measuring temporal parameters of VFSS. The proposed ResNet3D variants outperformed the existing models in detecting all phases of the swallowing process. Through the clinical application of this automatic model,, temporal analysis of VFSS will be easier and more accurate.

Videofluoroscopic swallowing study

deep learning

ResNet3D

Dysphagia, defined as difficulty in swallowing, is caused by neurological, muscular, anatomical, or psychological factors [1–3]. As dysphagia can lead to serious complication such as malnutrition, dehydration, aspiration pneumonia, and choking, it is important to accurately diagnose dysphagia [4–6].

Several diagnostic methods have been used for detecting dysphagia. Videofluoroscopic swallowing studies (VFSS) and fiberoptic endoscopic evaluation of swallowing is the gold standard for dysphagia evaluation. VFSS not only provides functional but also structural information regarding the swallowing process and is widely used in clinical practice [7]. To achieve a more objective evaluation, various assessment tools and sequential temporal analysis for VFSS are used for both clinical and research purposes [8–12]. However, this is time-consuming and complicated for clinicians to analyze [13]. To overcome these limitations, some studies using deep learning for the analysis of VFSS videos have been demonstrated [13–17]. For spatial analysis, two studies utilized deep learning to automatically detect hyoid bone movement and airway invasion [13, 17]. Some previous studies have applied deep learning for the temporal analysis of VFSS. However, most of them focused on detecting one specific phase, such as the pharyngeal phase or pharyngeal delay time [14–16].

Therefore, the purpose of this study was to develop an automatic model for measuring various temporal parameters of the VFSS during the overall swallowing process using deep learning. We also aimed to verify the performance of the proposed deep learning model compared to existing models.

Subjects

Patients who complained of dysphagia or were suspected of having dysphagia underwent VFSS at Korea University Guro Hospital from September 2020 to September 2021 and were consecutively recruited for this study. In total, 594 VFSS videos were retrospectively reviewed. The exclusion criteria were as follows: (1) patients younger than 19 years of age; (2) patients unable to progress from the pharyngeal phase to the esophageal phase because full analysis of the temporal parameters in these cases was not possible; (3) incompletely recorded video; and (4) when the contrast of the video was too low to identify anatomic structures. Based on the exclusion criteria, 47 videos were excluded from the study. Consequently, 547 ground-truth data samples were collected. This study was approved by the Institutional Review Board of the Korea University Guro Hospital (IRB No. 2021GR0568).

Videofluoroscopy swallowing study analysis

The VFSS was performed by a single rehabilitation physician. The participants were seated in an upright position and swallowed with barium-mixed materials. The lateral view of the head and neck region was recorded at a frequency of 15 frames per second (FPS) using the radio-fluoroscopy system Sonialvision G4 (Shimadzu Medical Systems and Equipment, Japan). Various amounts and viscosities of materials were used for study and videos of swallowing 2 mL of thin liquid were used. The thin liquid was then mixed with barium (35% w/v). Two experienced rehabilitation physicians independently analyzed the VFSS video clips. The intraclass coefficient was 0.999 (p-value < 0.001). If any disagreements occur between the two clinicians, a consensus was reached through discussion. The analysis process was conducted as follows: (1) extraction of the first subswallow video, (2) major event labeling, and (3) temporal parameter measurement.

Extracting The First Sub-swallow Video

If there were multiple subswallows, only the first sub-swallow video was extracted. We set the first frame as the starting point of the oral phase, and the last frame as the point between the swallow rest and the start of the next subswallow.

Major Event Labeling

As described in Fig. 1, major event labeling was performed in a manner similar to the ASPEKT method devised by Steele et al. [8]. The adjusted definitions of major events are as follows:

Start of oral phase

This event is defined as the time when the bolus first enters the oral cavity. This event is also the first frame of the video clip used for temporal analysis.

Bolus past mandible

This is the first frame in which the leading edge of the bolus touches or crosses the ramus of the mandible. If the two mandibular lines did not overlap, the midpoint between the upper and lower mandibular lines was considered as the reference point.

Burst of hyoid bone

This is the first frame when the hyoid bone starts jumping anterosuperiorly.

Laryngeal vestibule closure (LVC)

This event is the first frame when the inferior surface of the epiglottis and arytenoid process contact. When LVC occurs completely, the air space in the laryngeal vestibule becomes invisible. When LVC occurs incompletely, the frame in which the arytenoid process and the inferior surface of the epiglottis that is the most approximate, is used.

Upper esophageal sphincter (UES) opening

The UES first opens as a bolus or air passes through it.

UES closure

This is the first frame in which a single point or part of the UES segment closes behind the bolus tail.

Laryngeal vestibule closure offset

This is the earliest frame in which the air space of the laryngeal vestibule becomes visible.

Temporal Parameter Measurement

Seven temporal parameters including oral phase duration pharyngeal delay time, pharyngeal response time, pharyngeal transit time, laryngeal vestibule closure reaction time, laryngeal vestibule closure duration and upper esophageal sphincter opening duration were measured using the labeled major events, similar to the ASPEKT method devised by Steele et al. [8]. (Fig. 1) Since fluoroscopy was projected at 15 FPS, using the frame number of the main events, the following temporal parameters could be calculated in milliseconds.

Development of automatic models

Figure 2 presents an overview of the proposed method for phase localization using deep learning.

Three-dimensional Convolutional Neural Network (3d-cnn)

CNN is a type of artificial neural network based on convolution operations. The convolution operation effectively extracts and combines local image features and is applied to the input image over multiple times each of which is called a convolutional layer. The two-dimensional (2D) CNNs are typically used to analyze 2D images in deep learning. To process video data, 3D-CNNs can be applied, which add the time dimension to 2D images for input processing, that is, video is processed as a temporal sequence of images. In this study, we adopted ResNet3D which is a type of 3D-CNN, as explained in the following section [18].

Models

The base architecture we used was ResNet3D-18 [18]. ResNet3D-18 has skip connections in which the input bypasses the intermediate layers and is fed directly to the output. Skip connections facilitate the training of deep neural networks. The model consists of multiple residual blocks containing several convolutional layers and one skip connection. The residual block is depicted in Fig. 3. We adopted three architectural variants of ResNet3D-18, which are described as follows: [18]

DEFAULT This variant use the default configuration of ResNet3D-18 with only changes in the number of input frames. In this study, we conducted experiments with an input frame length set to either 7 or 13. The label for the input was set to that of the center of the input window. For example, the label of the input of seven frames is set to one if the 4th frame of the input is in the phase of interest, and zero otherwise.

FILL-16 In this variant, we sampled either seven or 13 successive frames and repeated those frames to obtain an input length of 16, which is the default input length for ResNet3D-18 [18]. For example, if we use a seven frame input, the input is repeated twice, and the first two frames are added at the end to make the input 16 frames. As previously mentioned, the label for the input was set as that of the center of the original input window.

BIDIRECTIONAL We proposed a new architectural variant of ResNet3D. Inspired by the bidirectional recurrent neural networks (RNN), we introduced a bidirectional structure in the ResNet3D which captures the forward and the backward stream features of a target frame as follows [19] (Fig. 4).

The bidirectional model combines the predictions from two separate ResNet3D models. Specifically, we used two models, denoted by $\mathbf{M}1$ and $\mathbf{M}2$, each of which takes $L$ frames as the input. Suppose that we would like to predict the label of the center frame of $2L-1$ frames. The input for $\mathbf{M}1$ is the first $L$ successive frames, that is, from the first frame to the center of the loaded frames. The input for $\mathbf{M}2$ is the last $L$ successive frames, i.e., from the center of the loaded frames to the last frame. Thus, we expect $\mathbf{M}1$ to learn forward temporal information, and $\mathbf{M}2$ to learn backward temporal information relative to the center frame. The outputs from $\mathbf{M}1$ and $\mathbf{M}2$ are combined and concatenated, and the combined vector was passed to a fully connected layer to yield the final prediction output.

Experimental Settings

Datasets The total number of VFSS video data was 547. The video dataset was split into training, validation, test sets with sizes of 444, 49, and 54 videos, respectively. The model input was red, green and blue (RGB) video frames which were resized to 224×224 pixels.

Baselines We compared our model with those from prior studies on VFSS. Lee et al. proposed a model based on VGG-16, which is a neural network with 16 weight layers widely used for image recognition [15, 20]. The I3D described in Lee et al. is a network which has been widely applied to action recognition and classification tasks [16, 21, 22]. We trained the baseline models on our dataset and optimized the hyperparameters.

Transfer learning We adopted the following transfer learning technique. Deep learning models must be trained on a large amount of data. However, it is difficult to collect large-scale medical data because of privacy issues or the small number of participants. Transfer learning retrains a model pretrained on large datasets of generic images or videos. All the models considered in this study used transfer learning. For example, VGG-16 was pretrained on ImageNet which contains 200 categories of 14,197, and 122 images [20, 21, 23]. I3D was pretrained on Kinetics-600 [21, 24]. The ResNet3D variants in this study were pretrained using Kinetics-700 [25]. The Kinetics datasets contained 650,000 human-action video clips. Each clip was annotated with a single action class and lasted approximately 10 seconds. Kinetics-600 and Kinetics-700 contained videos of 600 and 700 action classes, respectively. The pretrained model yielded substantially higher performance than those that were not pretrained.

Performance Evaluation

The accuracy, F1 score, average precision (AP) were calculated to comprehensively evaluate the temporal localization of the phases in the test dataset.

$$accuracy= \frac{the number of correct prediction}{the number of prediction}$$

$$recall= \frac{the number of positive prediction on the positive frames (=TP) }{the number of positive frames}$$

$$precision= \frac{the number of positive frames predicted positive}{the number of positive prediction}$$

$$F1 score= \frac{2}{\frac{1}{recall}+\frac{1}{precision}}= \frac{TP}{TP+\frac{1}{2}\left(the number of incorrect predicton\right)}$$

True positive (TP) denotes the number of positive predictions fort the positive frames. AP is the area under the precision-recall graph, and the higher the AP, the better the upper bound of 1. AP is a widely used metric for the temporal action localization task in computer vision and is particularly relevant to cases with class imbalance, that is, if only a small fraction of the given frames are the phase to be detected, which was our case.

Among the 594 participants, 47 were excluded based on the exclusion criteria. Finally, 547 subjects (71.8 ± 12.7 years, 337 males and 216 females) were included to develop automatic models. Stroke is the most common cause of dysphagia. The causes of dysphagia and penetration and aspiration score are described in Table 1 [26].

Table 1

Causes of dysphagia and Penetration and Aspiration Scale Values are number of patients
Cause of dysphagia
Stroke	207
Brain tumor	21
Traumatic brain injury	20
Parkinson’s disease	17
Cervical myelopathy	14
Head and neck cancer	12
Guillain-Barre syndrome	10
Encephalitis	9
Epilepsy	8
Esophageal cancer	7
Dementia	6
Motor neuron diseases	4
Myopathy	4
Cranial nerve lesion	3
Neuromuscular junctional disorder	1
Other medical condition	191
Unknown	13
Penetration Aspiration Scale
1	406
2	31
3	45
4	1
5	10
6	11
7	8
8	34

The temporal parameters and numbers of frame corresponding to each swallowing phase duration are summarized in Table 2.

Table 2

Swallowing phase durations and number of frames
Phase	Phase duration	Number of Frames
Oral Phase Duration	1.844 ± 2.281	27.658 ± 34.215
Pharyngeal Delay Time	1.387 ± 2.857	20.809 ± 42.856
Pharyngeal Response Time	0.677 ± 0.137	10.148 ± 2.060
Pharyngeal Transit Time	1.978 ± 2.880	29.676 ± 43.195
LVC Reaction time	0.302 ± 0.112	4.526 ± 1.687
LVC Duration	0.676 ± 0.266	10.053 ± 3.987
UES Opening Duration	0.432 ± 0.116	6.483 ± 1.746
Values are mean ± standard deviation
LVC laryngeal vestibule closure; UES, upper esophageal sphincter

We repeated the experiments for all phases for every model 20 times, averaged the results, and rounded off the numbers to three decimal places. The ResNet3D variants outperformed all baseline models in every phase of swallowing for the three metrics (Table 3). The architectural variants with the highest performance in each swallowing phase varied, as listed in Table 4. The best performing window sizes for the proposed variants are 7.

Table 3

Performance results of models
Model	Metric
Model	Accuracy	F1 score	Average precision
Oral phase duration
VGG	0.898$\pm 0.010$	0.832$\pm 0.016$	0.729$\pm 0.022$
I3D	0.918$\pm 0.014$	0.866$\pm 0.020$	0.775$\pm 0.031$
ResNet3D variation	0.92$\pm 0.009$*	0.87$\pm 0.014$*	0.779$\pm 0.022$*
Pharyngeal delay time
VGG	0.837$\pm 0.033$	0.644$\pm 0.158$	0.572$\pm 0.079$
I3D	0.861$\pm 0.008$	0.697$\pm 0.025$	0.633$\pm 0.021$
ResNet3D variation	0.864$\pm 0.007$*	0.703$\pm 0.210$*	0.640$\pm 0.018$*
Pharyngeal reaction time
VGG	0.94$\pm 0.007$	0.817$\pm 0.015$	0.687$\pm 0.022$
I3D	0.976$\pm 0.004$	0.915$\pm 0.013$	0.853$\pm 0.022$
ResNet3D variation	0.978$\pm 0.002$*	0.924$\pm 0.007$*	0.865$\pm 0.012$*
Pharyngeal transit time
VGG	0.858$\pm 0.009$	0.809$\pm 0.017$	0.766$\pm 0.012$
I3D	0.866$\pm 0.010$	0.813$\pm 0.018$	0.786$\pm 0.013$*
ResNet3D variation	0.869$\pm 0.010$*	0.822$\pm 0.018$*	0.786$\pm 0.014$*
LVC reaction time
VGG	0.946$\pm 0.013$	0.668$\pm 0.042$	0.48$\pm 0.043$
I3D	0.979$\pm 0.003$	0.818$\pm 0.043$	0.688$\pm 0.052$
ResNet3D variation	0.98$\pm 0.003$*	0.837$\pm 0.031$*	0.714$\pm 0.041$*
LVC duration
VGG	0.957$\pm 0.005$	0.859$\pm 0.012$	0.752$\pm 0.019$
I3D	0.963$\pm 0.005$	0.867$\pm 0.014$	0.769$\pm 0.023$
ResNet3D variation	0.966$\pm 0.007$*	0.875$\pm 0.035$*	0.787$\pm 0.041$*
UES Opening Duration
VGG	0.965$\pm 0.008$	0.844$\pm 0.028$	0.727$\pm 0.042$
I3D	0.976$\pm 0.007$	0.868$\pm 0.055$	0.779$\pm 0.065$
ResNet3D variation	0.981$\pm 0.003$*	0.901$\pm 0.013$*	0.823$\pm 0.022$*
Values are mean$\pm \text{s}\text{t}\text{a}\text{n}\text{d}\text{a}\text{r}\text{d} \text{d}\text{e}\text{v}\text{i}\text{a}\text{t}\text{i}\text{o}\text{n}$
LVC, laryngeal vestibule closure; UES, upper esophageal sphincter
*best result of three models

Table 4

ResNet3D architectural variants of best performance for each phase
Phase	Variants
Oral Phase Duration	FILL-16
Pharyngeal Delay Time	BIDIRECTIONAL
Pharyngeal Response Time	DEFAULT
Pharyngeal Transit Time	BIDERECTIONAL
LVC reaction time	BIDERECTIONAL
LVC duration	BIDERECTIONAL
UES opening duration	BIDERECTIONAL
LVC, laryngeal vestibule closure; UES upper esophageal sphincter

Our study successfully suggested an automatic model that can perform a temporal analysis of the entire swallowing phase using VFSS data. This is particularly valuable in that it can be easily applied in clinical practice because the overall swallowing phase was analyzed using realistic datasets without special manipulation.

VFSS is commonly used in dysphagia assessment and subjective visuoperception analysis by clinician is conducted as clinical purpose other than research [27, 28]. However, adequate intra- and inter-rater reliability cannot always be ensured in visuoperception analysis [29, 30]. If the quantified objective value is obtained from the test and the clinician uses it for VFSS analysis, the reliability will be improved. Quantitative temporal measurements of swallowing events can provide information regarding swallowing dynamics and the coordination of swallowing events. Specific swallow event delays were correlated with specific disabilities or stroke lesions in patients. Patients with decreased cognitive function commonly show oral phase delay, and those with brainstem stroke may show prolonged upper esophageal opening duration [31–33]. However, temporal analysis of VFSS images of all patients is very time-consuming and difficult to perform during the overall swallowing process. If it is possible to automatically detect and quantify the lengths of all swallowing phases, as in our study, clinicians may quickly and precisely identify which swallowing phase is problematic. In addition, it makes it easier for clinicians to compare the changes in dysphagia in patients before and after.

Although some studies using deep learning for temporal analysis of VFSS have been presented, most of these studies focused on detecting a specific phase, such as pharyngeal phase or pharyngeal delay time [14–16]. To the best of our knowledge, this is the first study to measure various temporal parameters throughout the swallowing process using deep learning. In addtion, deep learning models that were previously used for automated temporal analysis of VFSS have limitations. A study that used the VGG model, which is one of the 2D-CNN, did not consider the temporal relationships of the frames, and it may not learn the important muscle movements that determine the phase or the event [15]. In contrast, our study used 3D convolution operations to capture the temporal relationships of the frames. Another study that used the I3D model did not classify whether a frame belonged to the phase of interest, but instead classified whether the input frames overlapped with the phase of interest [16]. However, our method can classify each frame by detecting the presence of a phase at the center of the input frame. Another study which has three stages for phase detection, required training over optical flow data, in addition to RGB data [14]. Meanwhile, our method only used RGB data and consisted of only one stage, which lowered the computational cost.

Our study was conducted using VFSS videos of patients examined in clinical practice. In these videos, the head and shoulders of the patients were included, as well as the oral and pharyngeal areas. During VFSS examination, the head and shoulders of patients may take various poses and sometimes move back and forth. Occasionally, the hands of the clinician or patient may appear. Moreover, some videos had low clarity owing to poor focus or contrast (Fig. 5). In reality, it is difficult for some patients to maintain proper posture or consistently maintain a fixed posture. In fact, the performance of the baseline models reproduced using our data was lower than that reported in the original papers [15–17] (Table 3). This is because the models were trained on different datasets. The anatomical extent recorded in the video clips used in previous studies was limited to the area around the neck, thus, it is important to train or evaluate the model with realistic datasets for its applicability to clinical practice.

Among these phases, pharyngeal delay time and LVC reaction time yielded relatively poor results. For example, these phases were different from others; for example, the terminal point of the phases sometimes preceded the initiation point. For example, in the pharyngeal delay time, the action of the burst of hyoid bone event typically follows the event of bolus past mandible, but sometimes the burst of hyoid bone event proceed the bolus past mandible event. This makes the training of the model more difficult.

This study has some limitations. We used a fluoroscopy machine with 15 FPS, unlike many previous studies, which used 30 FPS machines. Although this allowed us to reduce the radiation dose to the patients, the results of the temporal analysis could have been less accurate. In addition, participants who could not progress from the pharyngeal phase to the esophageal phase due to severe upper esophageal sphincter dysfunction were excluded from this study; therefore, these cases would be difficult to analyze using this model. In addition, only videos in which the patients swallowed 2 cc of liquid were analyzed. There was no residue in the videos since 2cc of the thin liquid was the first to be examined among the materials of various consistencies and volumes. Therefore, only videos swallowing a 2cc thin liquid were used for the development of a more accurate model. The accuracy of the model can be improved if further studies include cases where previous residues remain as well as varying volumes and viscosities of the materials.

For future work using deep learning models for VFSS, we plan to develop models for multi-labeled event detection instead of phase localization. Event detection indicates that the model detects important changes in anatomic structures that determine the phases of the swallowing process. For example, if a model can detect the events of bolus past mandible, and burst of hyoid bone, it could be easier to identify the pharyngeal delay time. This is because it is easier to detect individual events involving movements than to detect a phase as a whole.

We demonstrated an automatic model that includes various temporal parameters of the VFSS. The proposed ResNet3D variants outperformed the existing models in detecting all phases of the swallowing process. Using this automatic model reduces the clinician labor and allows for a faster assessment of dysphagia.

VFSS: Videofluoroscopic swallowing study; LVC: Laryngeal vestibule closure; UES: upper esophageal sphincter; FPS: frame per second; 3D: three-dimensional; CNN: Convolutional Neural Network; RNN: recurrent neural networks; RGB: red, green and blue; AP: average precision; TP: true positive

Acknowledgments

Not applicable

Authors’ contributions

Conception and design: SB and SY. Data collection: JK, JB and SY. Data analysis SJ, JK and SY. Interpretation of the results and statistics: SJ and SB. Writing manuscript: SJ and JK. Review of writing: SB and SY. All authors read and approved the final manuscript.

Funding: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ICT Creative Consilience program (IITP-2022-2020-0-01819) supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation)

Availability of data and materials

The datasets for this study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

This retrospective study was approved by the Korea University Guro Hospital Institutional Review Board (IRB No. 2021GR0568).

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Roden DF, Altman KW. Causes of dysphagia among different age groups: a systematic review of the literature. Otolaryngol Clin North Am. 2013;46(6):965-87. https://doi.org/10.1016/j.otc.2013.08.008.
Sura L, Madhavan A, Carnaby G, Crary MA. Dysphagia in the elderly: management and nutritional considerations. Clin Interv Aging. 2012;7:287-98. https://doi.org/10.2147/cia.S23404.
Wieseke A, Bantz D, Siktberg L, Dillard N. Assessment and early diagnosis of dysphagia. Geriatr Nurs. 2008;29(6):376-83. https://doi.org/10.1016/j.gerinurse.2007.12.001.
Langmore SE, Terpenning MS, Schork A, Chen Y, Murray JT, Lopatin D, et al. Predictors of aspiration pneumonia: how important is dysphagia? Dysphagia. 1998;13(2):69-81. https://doi.org/10.1007/pl00009559.
Martin BJ, Corlew MM, Wood H, Olson D, Golopol LA, Wingo M, et al. The association of swallowing dysfunction and aspiration pneumonia. Dysphagia. 1994;9(1):1-6. https://doi.org/10.1007/bf00262751.
Jones CA, Colletti CM, Ding MC. Post-stroke Dysphagia: Recent Insights and Unanswered Questions. Curr Neurol Neurosci Rep. 2020;20(12):61. https://doi.org/10.1007/s11910-020-01081-z.
Logemann JA. Evaluation and treatment of swallowing disorders. 2nd ed. Austin, Tex.: PRO-ED; 1998.
Steele CM, Peladeau-Pigeon M, Barbon CAE, Guida BT, Namasivayam-MacDonald AM, Nascimento WV, et al. Reference Values for Healthy Swallowing Across the Range From Thin to Extremely Thick Liquids. J Speech Lang Hear Res. 2019;62(5):1338-63. https://doi.org/10.1044/2019_jslhr-s-18-0448.
Waito AA, Steele CM, Peladeau-Pigeon M, Genge A, Argov Z. A Preliminary Videofluoroscopic Investigation of Swallowing Physiology and Function in Individuals with Oculopharyngeal Muscular Dystrophy (OPMD). Dysphagia. 2018;33(6):789-802. https://doi.org/10.1007/s00455-018-9904-9.
Kang BS, Oh BM, Kim IS, Chung SG, Kim SJ, Han TR. Influence of aging on movement of the hyoid bone and epiglottis during normal swallowing: a motion analysis. Gerontology. 2010;56(5):474-82. https://doi.org/10.1159/000274517.
Curtis DJ, Cruess DF, Dachman AH, Maso E. Timing in the normal pharyngeal swallow. Prospective selection and evaluation of 16 normal asymptomatic patients. Invest Radiol. 1984;19(6):523-9. https://doi.org/10.1097/00004424-198411000-00009.
Kendall KA, Leonard RJ. Hyoid movement during swallowing in older patients with dysphagia. Arch Otolaryngol Head Neck Surg. 2001;127(10):1224-9. https://doi.org/10.1001/archotol.127.10.1224.
Zhang Z, Coyle JL, Sejdić E. Automatic hyoid bone detection in fluoroscopic images using deep learning. Sci Rep. 2018;8(1):12310. https://doi.org/10.1038/s41598-018-30182-6.
Lee JT, Park E, Jung TD. Automatic Detection of the Pharyngeal Phase in Raw Videos for the Videofluoroscopic Swallowing Study Using Efficient Data Collection and 3D Convolutional Networks (†). Sensors (Basel). 2019;19(18). https://doi.org/10.3390/s19183873.
Lee KS, Lee E, Choi B, Pyun SB. Automatic Pharyngeal Phase Recognition in Untrimmed Videofluoroscopic Swallowing Study Using Transfer Learning with Deep Convolutional Neural Networks. Diagnostics (Basel). 2021;11(2). https://doi.org/10.3390/diagnostics11020300.
Lee JT, Park E, Hwang JM, Jung TD, Park D. Machine learning analysis to automatically measure response time of pharyngeal swallowing reflex in videofluoroscopic swallowing study. Sci Rep. 2020;10(1):14735. https://doi.org/10.1038/s41598-020-71713-4.
Lee SJ, Ko JY, Kim HI, Choi S-I. Automatic Detection of Airway Invasion from Videofluoroscopy via Deep Learning Technology. Applied Sciences. 2020;10(18):6179.
Hara K, Kataoka H, Satoh Y, editors. Learning spatio-temporal features with 3d residual networks for action recognition. Proceedings of the IEEE international conference on computer vision workshops; 2017.
Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans Signal Process. 1997;45(11):2673-81.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. 2014.
Carreira J, Zisserman A, editors. Quo vadis, action recognition? a new model and the kinetics dataset. proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017.
Ioffe S, Szegedy C, editors. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning; 2015: PMLR.
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L, editors. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition; 2009: Ieee.
Carreira J, Noland E, Banki-Horvath A, Hillier C, Zisserman A. A short note about kinetics-600. arXiv preprint arXiv:1808.01340. 2018.
Smaira L, Carreira J, Noland E, Clancy E, Wu A, Zisserman A. A short note on the kinetics-700-2020 human action dataset. arXiv preprint arXiv:2010.10864. 2020.
Rosenbek JC, Robbins JA, Roecker EB, Coyle JL, Wood JL. A penetration-aspiration scale. Dysphagia. 1996;11(2):93-8. https://doi.org/10.1007/bf00417897.
Swan K, Cordier R, Brown T, Speyer R. Visuoperceptual Analysis of the Videofluoroscopic Study of Swallowing: An International Delphi Study. Dysphagia. 2021;36(4):595-613. https://doi.org/10.1007/s00455-020-10174-3.
Lee JW, Randall DR, Evangelista LM, Kuhn MA, Belafsky PC. Subjective Assessment of Videofluoroscopic Swallow Studies. Otolaryngol Head Neck Surg. 2017;156(5):901-5. https://doi.org/10.1177/0194599817691276.
Scott A, Perry A, Bench J. A study of interrater reliability when using videofluoroscopy as an assessment of swallowing. Dysphagia. 1998;13(4):223-7. https://doi.org/10.1007/pl00009576.
Baijens L, Barikroo A, Pilz W. Intrarater and interrater reliability for measurements in videofluoroscopy of swallowing. Eur J Radiol. 2013;82(10):1683-95. https://doi.org/10.1016/j.ejrad.2013.05.009.
Seçil Y, Arıcı Ş, İncesu TK, Gürgör N, Beckmann Y, Ertekin C. Dysphagia in Alzheimer's disease. Neurophysiol Clin. 2016;46(3):171-8. https://doi.org/10.1016/j.neucli.2015.12.007.
Moon HI, Pyun SB, Kwon HK. Correlation between Location of Brain Lesion and Cognitive Function and Findings of Videofluoroscopic Swallowing Study. Ann Rehabil Med. 2012;36(3):347-55. https://doi.org/10.5535/arm.2012.36.3.347.
Jo SY, Hwang JW, Pyun SB. Relationship Between Cognitive Function and Dysphagia After Stroke. Ann Rehabil Med. 2017;41(4):564-72. https://doi.org/10.5535/arm.2017.41.4.564.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Model	Metric
Model	Accuracy	F1 score	Average precision
Oral phase duration
VGG	0.898\(\pm 0.010\)	0.832\(\pm 0.016\)	0.729\(\pm 0.022\)
I3D	0.918\(\pm 0.014\)	0.866\(\pm 0.020\)	0.775\(\pm 0.031\)
ResNet3D variation	0.92\(\pm 0.009\)*	0.87\(\pm 0.014\)*	0.779\(\pm 0.022\)*
Pharyngeal delay time
VGG	0.837\(\pm 0.033\)	0.644\(\pm 0.158\)	0.572\(\pm 0.079\)
I3D	0.861\(\pm 0.008\)	0.697\(\pm 0.025\)	0.633\(\pm 0.021\)
ResNet3D variation	0.864\(\pm 0.007\)*	0.703\(\pm 0.210\)*	0.640\(\pm 0.018\)*
Pharyngeal reaction time
VGG	0.94\(\pm 0.007\)	0.817\(\pm 0.015\)	0.687\(\pm 0.022\)
I3D	0.976\(\pm 0.004\)	0.915\(\pm 0.013\)	0.853\(\pm 0.022\)
ResNet3D variation	0.978\(\pm 0.002\)*	0.924\(\pm 0.007\)*	0.865\(\pm 0.012\)*
Pharyngeal transit time
VGG	0.858\(\pm 0.009\)	0.809\(\pm 0.017\)	0.766\(\pm 0.012\)
I3D	0.866\(\pm 0.010\)	0.813\(\pm 0.018\)	0.786\(\pm 0.013\)*
ResNet3D variation	0.869\(\pm 0.010\)*	0.822\(\pm 0.018\)*	0.786\(\pm 0.014\)*
LVC reaction time
VGG	0.946\(\pm 0.013\)	0.668\(\pm 0.042\)	0.48\(\pm 0.043\)
I3D	0.979\(\pm 0.003\)	0.818\(\pm 0.043\)	0.688\(\pm 0.052\)
ResNet3D variation	0.98\(\pm 0.003\)*	0.837\(\pm 0.031\)*	0.714\(\pm 0.041\)*
LVC duration
VGG	0.957\(\pm 0.005\)	0.859\(\pm 0.012\)	0.752\(\pm 0.019\)
I3D	0.963\(\pm 0.005\)	0.867\(\pm 0.014\)	0.769\(\pm 0.023\)
ResNet3D variation	0.966\(\pm 0.007\)*	0.875\(\pm 0.035\)*	0.787\(\pm 0.041\)*
UES Opening Duration
VGG	0.965\(\pm 0.008\)	0.844\(\pm 0.028\)	0.727\(\pm 0.042\)
I3D	0.976\(\pm 0.007\)	0.868\(\pm 0.055\)	0.779\(\pm 0.065\)
ResNet3D variation	0.981\(\pm 0.003\)*	0.901\(\pm 0.013\)*	0.823\(\pm 0.022\)*
Values are mean\(\pm \text{s}\text{t}\text{a}\text{n}\text{d}\text{a}\text{r}\text{d} \text{d}\text{e}\text{v}\text{i}\text{a}\text{t}\text{i}\text{o}\text{n}\)
LVC, laryngeal vestibule closure; UES, upper esophageal sphincter
*best result of three models

Application of Deep Learning Technology for Temporal Analysis of Videofluoroscopic Swallowing Studies

Status:

Version 1

Abstract

Figures

Introduction

Methods

Extracting The First Sub-swallow Video

Major Event Labeling

Temporal Parameter Measurement

Three-dimensional Convolutional Neural Network (3d-cnn)

Models

Experimental Settings

Performance Evaluation

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1