Spectral Organ Fingerprints For Machine Learning-Based Intraoperative Tissue Classification With Hyperspectral Imaging

doi:10.21203/rs.3.rs-1328439/v1

Download PDF

Research Article

Spectral Organ Fingerprints For Machine Learning-Based Intraoperative Tissue Classification With Hyperspectral Imaging

https://doi.org/10.21203/rs.3.rs-1328439/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method’s current lack of robustness and generalizability. Specifically, it had been unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9,059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95 %). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decision making and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Category for submission

Experimental study

Level of Evidence

Not applicable. Translational animal science. Original article.

hyperspectral imaging

multispectral imaging

organ classification

tissue classification

organ identification

tissue identification

translational research

porcine model

machine learning

deep learning

linear mixed model

surgery

surgical data science

Discrimination of tissue conditions, pathologies and critical structures from healthy surrounding tissue during surgery can be challenging given the fact that different body tissues appear similar to the human eye. While conventional intraoperative imaging is limited by mimicking the human eye, hyperspectral imaging (HSI) removes this arbitrary restriction of recording only red, green and blue (RGB) colors. HSI works by assigning each pixel of a conventional two-dimensional digital image a third dimension of spectral information. The spectral information contains the wavelength-specific reflectance intensity of every pixel. This results in a three-dimensional datacube with two spatial dimensions (x, y) and a third spectral dimension (λ). HSI has found application in diverse fields such as geology and maritime studies, agriculture, food industry, automated waste sorting ^1,2 and has recently been used during a NASA space mission on Mars.

Over the last few years, there have been extensive efforts to implement HSI technology in healthcare. Examples of potential future clinical applications comprise the objective evaluation of tissue oxygenation and blood perfusion ^3-6, inflammation and sepsis ⁷ or malignancy ⁸ as well as computer-assisted decision-making and automated organ identification ⁹. These have the potential to support future developments such as intraoperative cognitive assistance systems or even automatization of robotic surgery. Despite the promising research, clinical translation of HSI-based automatic tissue differentiation has not yet been achieved. This may be attributed to a current lack in robustness and generalizability, which are the most important requirements for clinical application. In this regard, several open research questions remain. Specifically, variability of HSI measurements may result from the inherent differences between multiple tissue types under observation (desired effect), but also from inter-subject variability or variability in image acquisition conditions (both undesired). We are not aware of any prior work that has systematically investigated this important topic and we ultimately aim to provide a thorough understanding of hyperspectral organ data, illustrate the potential of HSI-based analyses and present solid baseline data that further studies can build upon.

For automatic tissue characterization based on HSI data, the following two properties are highly desirable: First, spectra corresponding to different organs should differ substantially from each other. And second, spectra of the same organ should be relatively constant across image acquisition conditions and individuals. With this in mind and given the gap in literature pointed out in the section above, the contribution of this work is threefold:

Spectral fingerprints: We present the first comprehensive analysis of spectral tissue properties for a wide range of physiological organs and tissue types. Based on 9,059 images of 46 pigs and 17,777 annotations, we generate specific spectral fingerprints for a total of 20 organs.
Variance analysis: We show that the greatest part of spectral variance can be explained by organ differences.
Machine learning-based organ and tissue classification with HSI: We demonstrate that a neural network can distinguish between organ classes with high accuracy (> 95%), suggesting that HSI has high potential for intraoperative organ and tissue discrimination

2.1 Different organs feature unique spectral fingerprints

This project provides insight on the spectral reflectance of 20 porcine organs in a total number of 9,059 images within 46 animals (Figure 1). Our data shows that different organs feature characteristic spectra, thus referred to as organ “fingerprints”. As seen in the gray pig-specific reflectance curves in Figure 1, variation in the spectral measurements may result not only from the organ, but also from the individuals and/or the specific measurement conditions. A key aim of this work was therefore to quantify the effect of the different sources of variation.

2.2 Spectral similarity between organs is heterogeneous

In order to illustrate HSI variability resulting from individuals and measurement conditions, t-distributed Stochastic Neighbor Embedding (t-SNE) ¹⁰ was applied to our data (Figure 2). It shows that while certain tissue types such as spleen and liver form highly isolated clusters, other organs such as stomach, pancreas and jejunum have a tendency to overlap, indicating lower distinguishability.

2.3 Organ is the most influential factor on the reflectance spectrum

To quantify the effect of different sources of variation, we applied linear mixed models on a highly standardized subset of data obtained from 11 pigs (within P36 to P46 as illustrated in Supplementary Figure 1). The analysis was performed at first for all organs (Figure 3) and subsequently stratified by organ (Figure 4). In the analysis for all organs, at each wavelength the proportion of explained variation ¹¹ in observed reflectance was decomposed into the components “organ”, “pig”, “angle”, “image” and “repetition”, where “angle” describes the proportion of variation explained by the angle between the organ surface and the camera optical axis, “image” describes the proportion of variation explained by different measurements taken from different organ positions in the same individual or variations in the annotated areas, and “repetition” describes the proportion of explained variation by multiple recordings of the same image under identical measurement conditions.

Our analysis suggests that the main influencing factor on HSI data variation across wavelengths was the factor “organ” with an average proportion of explained variability of 83.1 %. The factor “image” explained 13.4 % of the variation on average while the other factors only explained negligible variation with 2.0 % for “pig”, 1.2 % for “angle” and 0.5 % for “repetition”. This suggests that HSI data is characteristic of organs much more than of the subjects under observation or other influencing factors. The percentage to which variance in reflectance was explained by the components was not constant, but varied slightly through different parts of the recorded electromagnetic spectrum.

When stratifying by organ, the variance in reflectance was decomposed into the same components except “organ”. According to Figure 4, “angle” and “repetition” explain a negligible portion of the variance in all organs. For “pig” and “image”, differences between organs are present. For organs where all lines are close to zero (e.g. spleen), there is essentially no heterogeneity in reflectance between different images and pigs, thus these organs show the most pronounced organ-characteristic spectral signatures. On the other hand, organs with greater levels of explained variance for the components “pig” and “image” consequently had less organ-characteristic spectral signatures, such as lung with a comparatively high mean value of 0.0054 for variation explained by “pig”. Organ classes with the highest cumulative levels of variance curves explained by factors other than “organ” and therefore the least organ-characteristic spectral signatures across observations were lung and skin (Supplementary Text 3 and Supplementary Table 1).

For some organs, such as the gallbladder, reflectance varied strongly between pigs (value for “pig” comparatively high), but little within a pig (value for “image“ comparatively low). Thus, reflectances measured for gallbladders were heterogeneous across individual pigs. On the other hand, for other organs, heterogeneity within pigs (i.e. between images of the same pig) was much larger (value for “image” high) than between pigs (value for “pig” low), e.g. for skin. Thus, reflectance measured for skin tends to be homogeneous across individual pigs, but a single image of skin may be unreliable due to the heterogeneity within one pig.

2.4 Machine learning can leverage spectral information to classify tissue with high accuracy

A deep learning-based approach was used to classify the annotations of 20 organ classes from the spectra presented above with an average accuracy of 95.4 % ± 3.6 % across pigs on a hold-out test set. Misclassification only occurred for 486 out of 9,895 annotations in the test set (Figure 5). While 16 out of the 20 organ classes were classified with an average sensitivity of 90 % across all pigs, the smallest average sensitivity across pigs was obtained for the organ classes gallbladder (74.0 %) and heart (73.9 %), which were on average across pigs most often confused with bladder and kidney, respectively.

Visual discrimination and evaluation of biological tissue is not trivial, as different tissues and body structures often appear similar to the human eye. Because conventional optical imaging during surgery only differentiates red, green and blue by mimicking human vision, its intraoperative benefit is sometimes limited. HSI, not being subject to this restriction and encompassing significantly more information, is an exceptional imaging modality with great potential for tissue identification and evaluation. Although its current use in medicine is on a constant rise, the full potential of this imaging modality has not been exploited. This may be attributed to open research questions concerning robustness and generalizability of HSI data.

Structural properties of tissue cause differences in spectral characteristics that might be significant enough for use in proper organ differentiation and other clinical applications. However, existing literature on spectral measurements has mainly focused on specific biological pigments such as hemoglobin, porphyrin and melanin ^12,13, and has hardly addressed the complexity of spectral characteristics across various tissues and organs. Moreover, past works have often focused on optical scattering instead of reflectance or absorbance and therefore provided data of limited practical applicability, have investigated ex-vivo material, featured low wavelength resolution or performed measurements with incompatible, incomparable and outdated technology that does not allow for comparison between different studies ¹⁴. Despite its necessity as a foundation for advanced studies, as of today, no systematic database or investigation of reflectance spectra for a variety of physiological organs in a larger cohort existed.

Categorical requirements for such a spectral medical HSI database serving as a reference work are precision, uniformity and comparability of the measuring device, which has been demanded by the HSI community in previous years ¹⁵. While in former decades HSI could not be found in medicine, there have been extensive efforts to implement this technology in healthcare over the last years. However, most of the initially developed HSI systems were self-made prototypes and homebuilt solutions from various institutions all over the world, varying in spectral resolution and range as well as utilized detectors and optical components ^15-25. While highly interesting insights for various medical applications were obtained with these provisional solutions, they were lacking standardization and reproducibility ¹⁵, rendering sustainable large clinical trials and systematic multicenter research impossible. A great variety of devices could be observed in terms of spectral resolution, detectors, dispersive devices and spectral regions covered by different devices reaching from 200 nm up to 2500 nm ¹⁵.

The HSI camera system used in this project is the first commercially available and medically certified system meeting the aforementioned demands. While previous and less standardized HSI systems were efficient for the investigation of specific and isolated research questions, reproducibility and generalizability of commercially available systems noticeably promoted an increase in research efforts regarding HSI. An indicator of these increased research efforts can be seen in the rise of the number of research projects over the last few years including animal studies with rats ²⁶ and pigs ^6,27-29, conference papers ^30,31, narrative reviews ^32-34 and other publications ^35,36. With this new system and its advantages, special focus has again been put on early clinical trials with explorative character ^1,37-52. However, there are novel possibilities that have not been exploited yet. These primarily include spectral characterization of biological tissue and the complementation of a large medical HSI database with machine learning and deep learning. Some studies have already spectrally characterized single aspects of biological tissue such as the differences between specific cancer entities and their related physiological tissue ⁵³. However, these studies were most often conducted with non-reproducible setups or sometimes done in-vitro without measuring tissue perfusion, which might be acceptable for specific bradytrophic tissues, but leads to limitations in applicability to the majority of typically well-perfused organs ⁵⁴. Moreover, most of the existing studies so far only highlight specific medical aspects and do not sufficiently broaden the general understanding of spectral tissue characteristics.

The principles of spectral tissue differentiation have already successfully been proven, however only in laparoscopic surgery with sparse multispectral information and, most importantly, in fewer organ classes ^55,56. The question driving the present study was whether these spectral differences would be strong enough to be detected by an HSI system and subsequently consistent enough to characterize organs and make organ differentiation feasible.

For the very first time, HSI was applied with the aim of (1) systematically characterizing spectral properties of different tissue types in a porcine model, (2) analyzing to which extent these spectra are influenced by organ or tissue type compared to undesired effects such as inter-subject variability and variations in image acquisition conditions and (3) demonstrating that automatic machine learning-based tissue classification even with an unusually high number of classes can be achieved with high accuracy. A total number of 20 different porcine organs were recorded with HSI. The resulting database comprises 9,059 recordings with 17,777 annotated organ regions.

Spectral fingerprints of these organs were extracted in Figure 1 and t-SNE was chosen to visually assess the distinguishability of the respective HSI spectra (Figure 2). While Euclidean distances have to be interpreted cautiously in illustrations from high-dimension reduction tools, clustering and overlap give a good hint at the differentiability of the underlying spectra. It was now essential to evaluate to what extent differences in reflectance could be attributed to the organ or alternatively to the individual pig or noise from other defined and undefined factors as this would determine the general utility of HSI data.

Linear mixed models could show that the largest proportion of the spectral reflectance variability was attributed to the factor “organ” instead of “pig”, “angle”, “image” and “repetition”. This suggests that contributions from inter-individual differences and image acquisition conditions were dominated by organ differences. While image acquisition conditions such as illumination were highly standardized, artificial over-standardisation was consciously avoided in order to still comply with conditions in the real operative room. Of the other factors accounting for spectral reflectance variability, “image” was the most relevant one, which indicates, that different regions on the same organ have spectral differences. Possible explanations for this finding are inhomogeneous distribution of connective tissues, blood vessels and fibrosis within each organ, different levels of contained blood volumes due to tension on the tissue surface or peristalsis. This insight explicitly for the influencing factor “image” is highly relevant when considering possible real-life intraoperative applications and trials, as it - depending on the depth of the analysis - implies the necessity to record different areas of the organ under investigation as we did.

A machine learning algorithm had an average accuracy of over 95 % in an independent test set for identifying organ classes on pre-annotated regions, making this work a solid proof of concept for automatic tissue classification with machine learning based on HSI data. It is to be considered that automatic semantic scene annotation, however, might still present another challenge.

The number of organ recordings is heterogeneous throughout the dataset, which is due to the fact that recordings were done during diverse additional compatible surgical experiments. Moreover, we only considered the median spectrum of one pre-annotated region in one recording, thus ignoring texture information that may further improve organ identification. Notably, excellent classification results could already be achieved despite limiting the neural network input to organ reflectance without texture information. Organs with similar cellular composition such as stomach and jejunum showed similar reflectance spectra, but could still be differentiated well. Misclassifications mainly occurred between bladder and gallbladder, kidney and heart or vena cava and bone.

Besides the investigation of physiological organs, the systematic investigation of pathological states is of likewise importance and need to include tissue ischemia, stasis, inflammation and malignancy. The fact that these unphysiological organ states cannot be purposely induced in patients for ethical reasons necessitates the use of a large animal model with human-like features and known spectral tissue properties and marked the reason for choosing a porcine model for the present study.

For proper interpretation of the results of this work, certain limitations inherent to HSI technology have to be taken into account. One limitation is the relatively low temporal resolution of current HSI systems with only one recording every 30 seconds and around seven seconds of recording time each. While more compact and faster devices are under development ⁵⁷, currently this limitation narrows down possible fields of application. However, it does not undermine the validity of the data presented in this work. In fields of application that require a higher temporal resolution, but not necessarily fine-grained wavelength resolution, multispectral imaging (MSI) offers a solution ^4,58. These applications might include near video-rate imaging and can most probably be substantially refined when taking insights from HSI research into consideration.

Another limitation of HSI is the generally short and wavelength-dependent penetration depth of light in biological tissue. Increasing penetration depths between 700 and 1,000 nm had to be taken into account when measuring tissue with a thickness of less than several millimeters such as the omentum. Therefore, it was ensured by visual inspection that the omentum was only measured at sites with sufficient thickness. Photoacoustics, as a technology that is able to penetrate more deeply into biological tissue, might help to yield additional information when used complementarily to HSI ⁵⁹.

Further limitations arise due to the spatial resolution of only 680 x 480 pixels (width x height). Organs with smaller surface areas, e.g. the gallbladder, were harder to annotate than others, since fewer pixels were available. Therefore, the sizes of annotated regions are imbalanced between organs with smaller and larger surface areas. However, small sizes of annotations were successfully compensated by choosing a large number of recordings. Besides the technological limitations, we did not investigate variability resulting from camera variations and the presented tissue identification relied on pre-annotated regions of interest (ROIs). Variability resulting from camera variations and performance on machine learning-based semantic identification will have to be addressed in future studies.

This work is the first to systematically investigate spectral properties and relations of organs within a large cohort of organ classes and individuals. By using a highly standardized approach, we were able to extract the spectral fingerprints for each organ and investigate factors influencing the spectral properties. We were able to provide evidence that the recorded tissues and not the individual animal or the recording conditions were the most influential factor for the electromagnetic spectrum, which is of utmost importance when trying to assess the possible value of HSI for medical applications. This study can be seen as a reference work paving the way for spectral organ evaluation, which requires precise knowledge of spectral characteristics of physiological tissue. Possible future applications based on these results include augmentation of computer-assisted decision-making, intraoperative cognitive assistance systems or even automatization of robotic surgery. It can be expected that our main finding of organ-dependent reflectance patterns will be confirmed in human data. To firmly establish HSI in clinical medicine, a translation of this study to human data will be essential.

4.1 Animal anaesthesia & surgical procedure

This animal study was approved by the Committee on Animal Experimentation of the regional council of Baden-Württemberg in Karlsruhe, Germany (G-161/18 and G-262/19). All animals used in the experimental laboratory were managed according to German laws for animal use and care, and according to the directives of the European Community Council (2010/63/EU) and ARRIVE guidelines ⁶⁰. Data from 46 pigs was included in the analyses.

Experimental animals were operated under general anaesthesia with extensive monitoring including invasive blood pressure measurement. Midline laparotomy was performed to access the abdominal cavity. Ligaments around the liver and the hepato-gastric ligament were dissected and visceral organs mobilized, including the removal of the coverage of the kidneys while carefully sparing vessels. Scissors, electrocautery and bipolar vessel-sealing devices were used. A suprapubic catheter was inserted into the bladder. After surgery, pigs were euthanized with a lethal dose of i.v. potassium chloride solution.

4.2 Hyperspectral Imaging

The hyperspectral datacubes were acquired with the TIVITA® Tissue system (Diaspective Vision GmbH, Pepelow, Germany), which is a push-broom scanning imaging system and the first commercially available hyperspectral camera for medicine. It provides a high spectral resolution in the visible as well as near-infrared (NIR) range from 500 nm to 995 nm in 5 nm steps resulting in 100 spectral bands. Its field of view contains 640 x 480 pixels with a spatial resolution of approximately 0.45 mm/pixel (Figure 6). The distance of the camera to the specimen is controlled via a red-and-green light targeting system. Six halogen lamps directly integrated into the camera system provide a uniform illumination. Recording takes around seven seconds.

4.3 Image acquisition, annotation and processing

Images were recorded with a distance of 50 ± 5 cm between camera and organs. In order to prevent distortions of the measured reflectance spectra due to stray light, the tissue recordings were made while lights in the operating room were switched off and curtains were closed. While the majority of pig recordings was done in a generic approach in order to accurately represent intraoperative reality, recordings for the mixed model analysis were done with a highly standardized protocol for a subset of 11 pigs (8 to 9 pigs per organ) (between P36 and P46 as indicated in Supplementary Text 1). This standardized protocol includes recordings of 3 repetitions of exactly the same surgical scene (“repetition” effect) from 3 different angles (“angle” effect) (perpendicular to the tissue surface, 25° from one side and 25° from the opposite side) for 4 different organ positions / situs / situations (“image” effect) resulting in a total of 36 recordings for each of the 20 organs (8 to 9 pigs per organ) in a total of 11 pigs. Recordings for bile fluid were performed when applied and soaked onto 5 stacked surgical compresses, ensuring that there is no influence from the background. For a more extensive overview of the dataset and a schematic recordings protocol for the standardized subset please refer to Supplementary Figure 1 and Supplementary Figure 2.

All of the 9,059 recorded images were sorted into the 20 respective organ folders and manually annotated resulting in 17,777 organ annotations (as several organs could be contained within one image). A precise annotation protocol can be found in Supplementary Text 4. Annotations were done by one medical expert and then verified by two other medical experts. In case of improper annotation, the annotation was redone collectively for that specific recording.

For t-SNE and the machine learning analysis, spectral information was previously L1-normalized at pixel-level for increased uniformity. All of the other analyses (including the structured model from Supplementary Text 2 and Supplementary Figure 3) required unprocessed reflectance values from the original datacubes. After annotation, the wavelength-specific annotation-wise median was automatically calculated over every pixel included in the annotation. These median annotation-wise spectra (previously either L1-normalized or not) represented the basic data format that all analyses in this paper were based upon. Calculation of the mean (and SD) integral of the organ reflectance curves of individual animals was performed to quantify overall brightness or amount of light that is reflected by the organ in relative units; greater values indicate greater reflectance intensity. Although this quantification of the overall level of the reflectance curve and therefore the area under the curve is influenced by the distance between camera and tissue, the standardization of this distance reduced this influence, rendering this integral a valuable information.

4.4 t-SNE

t-distributed Stochastic Neighbor Embedding (t-SNE) ¹⁰ is a machine learning method commonly used to reduce the number of dimensions of high-dimensional data and was used to visualize the characteristic reflectance spectra of each pig organ. This non‐linear multi-dimensionality reduction tool has already proven valuable for the analysis of HSI and mass spectrometry data ⁶¹ and was chosen for visualization as it has shown particular promise for biological samples in the past ^62,63. The algorithm aims at modelling manifolds of high‐dimensional data, and produces low‐dimensional embeddings that are optimized for preserving the local neighbourhood structure of the high-dimensional manifold ¹⁰. In comparison to linear methods like PCA ⁶⁴ and LDA ⁶⁵, t-SNE preserves more relevant structures of datasets that have non-linear features. For these reasons, t-SNE was used for dimensionality reduction.

Before optimizing the parameters of t-SNE, the dataset was prepared in the following manner: One characteristic reflectance spectrum was obtained for each annotation by calculating the median spectra from the (previously on pixel-level L1-normalized) spectra of all pixels in the annotation. Consequently, each data point represents the reflectance of one organ in one image of one pig. The two-dimensional visualization of the reflectance spectrum of the complete dataset was optimized by performing a random search of the following parameters:

Parameter 1: The early exaggeration, which controls how tight natural clusters in the original space are in the embedded space and how much space will be between them. 50 random integer values were sampled in the range [5; 100].
Parameter 2: The learning rate, which is used in the optimization process. 100 random integer values were sampled in the range [10; 1000].
Parameter 3: The perplexity, which is related to the number of nearest neighbors for each data point to be considered in the optimization. 50 equidistant integer values were sampled uniformly in the range [2; 100].

The early exaggeration was the first parameter optimized by visual inspection of the two-dimensional representation of the dataset. The learning rate was then optimized in the same manner while keeping the early exaggeration constant. Subsequently, the perplexity was optimized by keeping the other two parameters constant. The optimal values for each of the parameters were 34 for the early exaggeration, 92 for the learning rate and 30 for the perplexity.

4.5 Linear mixed models

Independent linear mixed models were used for an explained variation analysis in order to evaluate the effect of the influencing factors on changes in the spectrum. The (proportion of) explained variance was obtained using the empirical decomposition of the explained variation in the variance components form of the mixed model ¹¹.

For the first approach, for each wavelength, an independent linear mixed model was fitted with fixed effects for “organ” and “angle” as well as random effects for “pig” and “image”. More precisely, for each wavelength the following model was fitted (suppressing the wavelength index):

for repetition k=1,…,3 of image j=1,…,n_i of pig i=1,…, 11 (with n_i the number of images of pig i ranging from 84 to 228 and ). is an intercept, is a row vector of length 19 indicating the organ of observation ijk (with arbitrary reference category “stomach”) and is a vector of corresponding fixed organ effects. Similarly, are fixed effects for angle (“25° from one side” and “25° from the opposite side” for reference category “perpendicular to the tissue surface”). and are random pig and image effects, respectively, assumed to be independently normal distributed with between pig variation and between image variation Residuals capture the variability between repeated recordings of the same image.

The proportion of variability in reflectance explained by each factor was derived as in ¹¹. “Repetition” depicts the residual variability, which is here the within image variability (i.e. across replications). 95 % pointwise confidence intervals based on parametric bootstrapping with 500 replications indicate the uncertainty in estimates.

For the second approach with stratification by organ, independent linear mixed models were fitted for each organ and wavelength with fixed effects for “angle” as well as a random effect for “pig” and “image”, i.e for each organ and wavelength the same model as given above was fitted excluding covariate “organ”. The explained variation of each factor was depicted ¹¹. “Repetition” depicts the residual variation, which is here the within image variability (i.e. across replications). 95 % pointwise confidence intervals based on parametric bootstrapping with 500 replications indicate the uncertainty in estimates. Curves were linearly interpolated if model fit was singular. All linear mixed model analyses were based on image-wise organ-specific median reflectance spectra that were obtained by calculating the median spectrum of all pixel spectra within one annotation.

4.6 Machine Learning

Prior to training the deep learning network, we systematically split the dataset comprising 46 pigs (9,059 images with 17,777 annotations) into a training dataset consisting of 38 pigs (3,766 images with 7,882 annotations) and a disjoint test set consisting of 8 pigs (5,293 images with 9,895 annotations) as indicated in Supplementary Figure 1. These 8 test pigs were randomly selected from the 11 standardized pigs (P36-P46) with the only criterion that every organ class is represented by at least one standardized pig in the test as well as in the training dataset. This criterion could not be fulfilled anymore when selecting more than 8 standardized pigs.

The hold-out test set was used only after the network architecture and all hyperparameters had been fixed. Leave-one-pig-out cross-validation was performed on the training dataset and the predictions on the left-out pig were aggregated for all 38 folds (46 minus 8) to yield the validation accuracy. The hyperparameters of the neural network were optimized in an extensive grid search such that the validation accuracy was maximized. Once the optimal hyperparameters were determined, we evaluated the classification performance on the hold-out test set by ensembling the predictions from all 38 networks (one for each fold) via computing the mean logits vector (the input values to the softmax function, see below) followed by the argmax operation to retrieve the final label for each annotation.

The deep learning-based classification was performed on the median spectra computed from the L1-normalized spectra of all pixels in the annotation masks resulting in 100-dimensional input feature vectors.

The deep learning architecture was composed of 3 convolutional layers (64 filters in the first, 32 in the second, and 16 in the third layer) followed by 2 fully connected layers (100 neurons in the first and 50 in the second layer). The activations of all five layers were batch normalized and a final linear layer was used to calculate the class logits. Each of the convolutional layers convolved the spectral domain with a kernel size of 5 and was followed by an average pooling layer with a kernel size of 2. The two fully connected layers zeroed out their activations with a dropout probability of . All non-linear layers used the Exponential Linear Unit (ELU) ⁶⁶ as activation function.

We chose this architecture as it provides a simple yet effective way to analyze the spectral information. The convolution operation acts on the local structure of the spectra and we used a relatively small kernel size and stacked 3 layers to increase the receptive field while being computationally efficient ⁶⁷. The two fully connected layers make a final decision based on the global context. The advantage of this approach is that it combines local and global information aggregation while still being computationally efficient since the entire network only uses 34,300 trainable weights.

The softmax function was used to provide the a posteriori probability for each class. We used the Adam optimizer (β₁= 0.9, β₂= 0.999) ⁶⁸ with an exponential learning rate decay (decay rate of and initial learning rate of ) and the multiclass cross-entropy loss function. In order to meet class imbalances, we included an optional weight of the loss function according to the number of training images per class and sampled instances for the batches either randomly or oversampled such that each organ class had the same probability of being sampled. Both design choices were investigated in the hyperparameter grid search.

We trained 10,000,000 samples per epoch for 10 epochs with a batch size of . In an extensive grid search, we determined the best-performing hyperparameters: dropout probability , learning rate , decay rate , batch size , a weighted loss function and no oversampling.

HSI hyperspectral imaging

MSI multispectral imaging

NASA National Aeronautics and Space Administration

ROI region of interest

SD standard deviation

t-SNE t-distributed Stochastic Neighbor Embedding

Acknowledgements

The authors thank Minu Tizabi (CAMI, DKFZ) for proofreading the manuscript and gratefully acknowledge the data storage service SDS@hd supported by the Ministry of Science, Research and the Arts Baden-Württemberg (MWK) and the German Research Foundation (DFG) through grant INST 35/1314-1 FUGG and INST 35/1503-1 FUGG. The present contribution is supported by the Helmholtz Association under the joint research school HIDSS4Health (Helmholtz Information and Data Science School for Health). This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (NEURAL SPICING, grant agreement No. [101002198]) as well as from the Willi Robert Pitzer foundation, from the Heidelberg Foundation for Surgery and from the RISE program by the German Academic Exchange Service (DAAD). Depicted images in the figures were drawn by the authors. The image of the TIVITA® Tissue system has been made available with kind approval from Diaspective Vision.

Author contributions

ASF, FN and BPMS had the original idea for the project. ASF and FN initiated the project. ASF, FN and CMH performed the initial review of existing literature and the planning. ASF, CMH, KK, BÖ, MD, BPMS and FN performed the surgeries. ASF, IC, BÖ and JO developed the Python codes for data organization and annotation. ASF, GS and SK annotated data. ASF, GS, BÖ, JO, SK, SS, LA, JS, LMH, TA and MW analyzed and interpreted data. LMH, TA, LA, SS, JS, MW, NS, AKS and ASF developed and implemented the statistical and machine learning-based data analysis strategy. LA provided the t-SNE analysis and the relevant manuscript passages. MW, NS and AKS provided the mixed model analysis, the structured model and the relevant manuscript passages. SS and JS provided the machine learning-based classification and the relevant manuscript passages. FN, LMH, HGK, KMH, KS and BPMS provided expert knowledge throughout the project. ASF, BÖ and FN wrote the manuscript. SS, JS, LA, MW, BPMS, HGK, KMH and LMH revised the manuscript. All authors have read and approved the final manuscript.

Data availability statement

Data and code that support the findings of this study are available from the corresponding authors upon reasonable request.

Competing Interests Statement

Authors state no conflict of interest.

Mascagni, P. et al. New intraoperative imaging technologies: Innovating the surgeon’s eye toward surgical precision. Journal of surgical oncology 118, 265–282, doi:10.1002/jso.25148 (2018).
Prasad, S. & Chanussot, J. Hyperspectral Image Analysis - Advances in Machine Learning and Signal Processing. (Springer, 2020).
Ayala, L. A. et al. 38–46 (Springer International Publishing).
Wirkert, S. J. et al. 134–141 (Springer International Publishing).
Barberio, M. et al. Quantitative fluorescence angiography versus hyperspectral imaging to assess bowel ischemia: A comparative study in enhanced reality. Surgery, doi:10.1016/j.surg.2020.02.008 (2020).
Nickel*, F. et al. Optimization of anastomotic technique and gastric conduit perfusion with hyperspectral imaging in an experimental model for minimally invasive esophagectomy. bioRxiv, 2021.2010.2003.462901, doi:10.1101/2021.10.03.462901 (2021).
Dietrich*, M. et al. Machine learning-based analysis of hyperspectral images for automated sepsis diagnosis, <https://arxiv.org/abs/2106.08445v1> (2021).
Wu, I. C. et al. Early identification of esophageal squamous neoplasm by hyperspectral endoscopic imaging. Sci Rep 8, 13797, doi:10.1038/s41598-018-32139-1 (2018).
Clancy, N. T., Jones, G., Maier-Hein, L., Elson, D. S. & Stoyanov, D. Surgical spectral imaging. Medical Image Analysis 63, 101699, doi:https://doi.org/10.1016/j.media.2020.101699 (2020).
van der Maaten, L. J. P. & Hinton, G. E. Visualizing High-Dimensional Data Using t-SNE. Journal of Machine Learning Research 9, 2579–2605 (2008).
Schreck, N. Empirical decomposition of the explained variation in the variance components form of the mixed model. bioRxiv, 2019.2012.2028.890061, doi:10.1101/2019.12.28.890061 (2019).
Thunell, S. Porphyrins, porphyrin metabolism and porphyrias. I. Update. Scand J Clin Lab Invest 60, 509–540, doi:10.1080/003655100448310 (2000).
Wilson, M. T. & Reeder, B. J. Oxygen-binding haem proteins. Exp Physiol 93, 128–132, doi:10.1113/expphysiol.2007.039735 (2008).
Jacques, S. L. Optical properties of biological tissues: a review. Physics in Medicine and Biology 58, R37-R61, doi:10.1088/0031-9155/58/11/r37 (2013).
Lu, G. & Fei, B. Medical hyperspectral imaging: a review. Journal of biomedical optics 19, 010901 (2014).
Afromowitz, M. A., Callis, J. B., Heimbach, D. M., DeSoto, L. A. & Norton, M. K. Multispectral imaging of burn wounds: a new clinical instrument for evaluating burn depth. IEEE Transactions on Biomedical Engineering 35, 842–850, doi:10.1109/10.7291 (1988).
Ferris, D. G. et al. Multimodal Hyperspectral Imaging for the Noninvasive Diagnosis of Cervical Neoplasia. Journal of Lower Genital Tract Disease 5, 65–72, doi:https://doi.org/10.1046/j.1526-0976.2001.005002065.x (2001).
Shah, S. A. et al. Cutaneous Wound Analysis Using Hyperspectral Imaging. BioTechniques 34, 408–413, doi:10.2144/03342pf01 (2003).
Bambery, K. R., Wood, B. R., Quinn, M. A. & McNaughton, D. Fourier Transform Infrared Imaging and Unsupervised Hierarchical Clustering Applied to Cervical Biopsies. Australian Journal of Chemistry 57, 1139–1143, doi:https://doi.org/10.1071/CH04137 (2004).
Akbari, H., Kosugi, Y., Kojima, K. & Tanaka, N. Detection and Analysis of the Intestinal Ischemia Using Visible and Invisible Hyperspectral Imaging. IEEE Transactions on Biomedical Engineering 57, 2011–2017, doi:10.1109/TBME.2010.2049110 (2010).
Akbari, H. et al. Hyperspectral imaging and quantitative analysis for prostate cancer detection. Journal of biomedical optics 17, 076005 (2012).
Mitra, K. et al. Indocyanine-green-loaded microballoons for biliary imaging in cholecystectomy. J Biomed Opt 17, 116025 (2012).
Rosas, J. G. & Blanco, M. A criterion for assessing homogeneity distribution in hyperspectral images. Part 2: Application of homogeneity indices to solid pharmaceutical dosage forms. Journal of Pharmaceutical and Biomedical Analysis 70, 691–699, doi:https://doi.org/10.1016/j.jpba.2012.06.037 (2012).
Li, Q., Wang, Y., Liu, H. & Chen, Z. in 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE). 15-17.
Kumar, S., Desmedt, C., Larsimont, D., Sotiriou, C. & Goormaghtigh, E. Change in the microenvironment of breast cancer studied by FTIR imaging. Analyst 138, 4058–4065, doi:10.1039/c3an00241a (2013).
Grambow, E. et al. Hyperspectral imaging for monitoring of perfusion failure upon microvascular anastomosis in the rat hind limb. Microvascular research 116, 64–70, doi:10.1016/j.mvr.2017.10.005 (2018).
Barberio, M. et al. HYPerspectral Enhanced Reality (HYPER): a physiology-based surgical guidance tool. Surgical Endoscopy 34, 1736–1744, doi:10.1007/s00464-019-06959-9 (2020).
Felli, E., Urade, T., Barberio, M., Felli, E. & Diana, M. Hyperspectral imaging of pig liver ischemia: a proof of concept, <https://www.airitilibrary.com/Publication/alDetailedMesh?docid=15610497-201912-201912180004-201912180004-117-121> (2019).
Tetschke, F. et al. Hyperspectral imaging for monitoring oxygen saturation levels during normothermic kidney perfusion. J. Sens. Sens. Syst. 5, 313–318, doi:10.5194/jsss-5-313-2016 (2016).
Holmer, A., Kaemmerer, P., Kulcke, A., Marotz, J. & Wahl, P. in AUTOMED.
Landro, M. D. et al. in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 1851-1854.
Gockel, I. et al. Möglichkeiten und Perspektiven der Hyperspektralbildgebung in der Viszeralchirurgie. Der Chirurg 91, 150–159, doi:10.1007/s00104-019-01016-6 (2020).
Goetze, E., Thiem, D. G. E., Gielisch, M., Al-Nawas, B. & Kämmerer, P. W. Digitalisierung und Ansätze künstlicher Intelligenz in der mikrovaskulär-rekonstruktiven Gesichtschirurgie. Der Chirurg 91, 216–221, doi:10.1007/s00104-019-01103-8 (2020).
Maier-Hein, L. et al. Intraoperative Bildgebung und Visualisierung. Der Onkologe 26, 31–43, doi:10.1007/s00761-019-00695-4 (2020).
Holmer, A. et al. Oxygenation and perfusion monitoring with a hyperspectral camera system for chemical based tissue analysis of skin and organs. Physiological measurement 37, 2064–2078, doi:10.1088/0967-3334/37/11/2064 (2016).
Markgraf, W., Feistel, P., Thiele, C. & Malberg, H. Algorithms for mapping kidney tissue oxygenation during normothermic machine perfusion using hyperspectral imaging. Biomedical Engineering / Biomedizinische Technik 63, 557, doi:https://doi.org/10.1515/bmt-2017-0216 (2018).
Barberio, M. et al. Hyperspectral based discrimination of thyroid and parathyroid during surgery. 4, 399, doi:https://doi.org/10.1515/cdbme-2018-0095 (2018).
Daeschlein, G. et al. Hyperspectral imaging as a novel diagnostic tool in microcirculation of wounds. Clinical hemorheology and microcirculation 67, 467–474, doi:10.3233/ch-179228 (2017).
Grambow, E. et al. Evaluation of peripheral artery disease with the TIVITA(R) Tissue hyperspectral imaging camera system. Clinical hemorheology and microcirculation 73, 3–17, doi:10.3233/ch-199215 (2019).
Herrmann, B. H. & Hornberger, C. Monte-Carlo Simulation of Light Tissue Interaction in Medical Hyperspectral Imaging Applications. Current Directions in Biomedical Engineering 4, 275–278, doi:https://doi.org/10.1515/cdbme-2018-0067 (2018).
Jansen-Winkeln, B., Holfert, N., Köhler, H., Chalopin, C. & Gockel, I. Bestimmung der idealen Anastomosenposition durch hyperspectrale Bildgebung. Z Gastroenterol 57, KV 93, doi:10.1055/s-0039-1695193 (2019).
Jansen-Winkeln, B. et al. Determination of the transection margin during colorectal resection with hyperspectral imaging (HSI). Int J Colorectal Dis 34, 731–739, doi:10.1007/s00384-019-03250-0 (2019).
Jansen-Winkeln, B. et al. Handnaht v. Stapler-Anastomose – Hyperspektralbetrachtung der Perfusion. Z Gastroenterol 57, KV 91, doi:10.1055/s-0039-1695191 (2019).
Köhler, H., Jansen-Winkeln, B., Chalopin, C. & Gockel, I. Hyperspectral imaging as a new optical method for the measurement of gastric conduit perfusion. Diseases of the Esophagus 32, 1–1, doi:10.1093/dote/doz046 (2019).
Köhler, H. et al. Evaluation of hyperspectral imaging (HSI) for the measurement of ischemic conditioning effects of the gastric conduit during esophagectomy. Surgical Endoscopy 33, 3775–3782, doi:10.1007/s00464-019-06675-4 (2019).
Langner, I. et al. Hyperspektralimaging demonstriert mikrozirkulatorische Effekte postoperativer Ergotherapie bei Patienten mit Morbus Dupuytren. Handchirurgie, Mikrochirurgie, plastische Chirurgie: Organ der Deutschsprachigen Arbeitsgemeinschaft fur Handchirurgie : Organ der Deutschsprachigen Arbeitsgemeinschaft fur Mikrochirurgie der Peripheren Nerven und Gefasse 51, 171-176, doi:10.1055/a-0916-8635 (2019).
Maktabi, M. et al. Tissue classification of oncologic esophageal resectates based on hyperspectral data. International journal of computer assisted radiology and surgery 14, 1651–1661, doi:10.1007/s11548-019-02016-x (2019).
Marotz, J. et al. Extended Perfusion Parameter Estimation from Hyperspectral Imaging Data for Bedside Diagnostic in Medicine. Molecules 24, 4164, doi:10.3390/molecules24224164 (2019).
Marotz, J., Siafliakis, A., Holmer, A., Kulcke, A. & Siemers, F. First results of a new hyperspectral camera system for chemical based wound analysis. Wound Medicine 10-11, 17–22, doi:https://doi.org/10.1016/j.wndm.2015.11.003 (2015).
Mohammed, R. A. A., Schäle, D., Hornberger, C. & Emmert, S. Detecting Signatures in Hyperspectral Image Data of Wounds: A Compound Model of Self- Organizing Map and Least Square Fitting. Current Directions in Biomedical Engineering 4, 419–422, doi:https://doi.org/10.1515/cdbme-2018-0100 (2018).
Sucher, R. et al. Hyperspectral Imaging (HSI) in anatomic left liver resection. International journal of surgery case reports 62, 108–111, doi:https://doi.org/10.1016/j.ijscr.2019.08.025 (2019).
Zimmermann, P., Scheibe, A., Marotz, J. & Wollina, U. Analysis of tissue oxygenation in chronic leg ulcers by combination of a multi-spectral camera and a hyper-spectral probe. Georgian medical news, 75–81 (2017).
Maktabi, M. et al. Tissue classification of oncologic esophageal resectates based on hyperspectral data. International journal of computer assisted radiology and surgery, doi:10.1007/s11548-019-02016-x (2019).
Filatova, S., Shcherbakov, I. & Tsvetkov, V. Optical properties of animal tissues in the wavelength range from 350 to 2600 nm. Journal of biomedical optics 22, 035009 (2017).
Zhang, Y. et al. Tissue classification for laparoscopic image understanding based on multispectral texture analysis. J Med Imaging (Bellingham) 4, 015001, doi:10.1117/1.JMI.4.1.015001 (2017).
Moccia, S. et al. Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy. IEEE Trans Biomed Eng 65, 2649–2659, doi:10.1109/TBME.2018.2813015 (2018).
Ayala, L. et al. Video-rate multispectral imaging in laparoscopic surgery: First-in-human application. arXiv:2105.13901 (2021). <https://arxiv.org/abs/2105.13901>.
Wirkert, S. J. et al. Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression. International journal of computer assisted radiology and surgery 11, 909–917, doi:10.1007/s11548-016-1376-5 (2016).
Gröhl, J. et al. Semantic segmentation of multispectral photoacoustic images using deep learning. arXiv:2105.09624 (2021). <https://arxiv.org/abs/2105.09624>.
Kilkenny, C., Browne, W., Cuthill, I. C., Emerson, M. & Altman, D. G. Animal research: reporting in vivo experiments: the ARRIVE guidelines. British journal of pharmacology 160, 1577–1579, doi:10.1111/j.1476-5381.2010.00872.x (2010).
Gardner, W., Cutts, S. M., Phillips, D. R. & Pigram, P. J. Understanding mass spectrometry images: complexity to clarity with machine learning. Biopolymers n/a, e23400, doi:https://doi.org/10.1002/bip.23400 (2020).
Kobak, D. & Berens, P. The art of using t-SNE for single-cell transcriptomics. Nature Communications 10, 5416, doi:10.1038/s41467-019-13056-x (2019).
Cieslak, M. C., Castelfranco, A. M., Roncalli, V., Lenz, P. H. & Hartline, D. K. t-Distributed Stochastic Neighbor Embedding (t-SNE): A tool for eco-physiological transcriptomic analysis. Marine Genomics 51, 100723, doi:https://doi.org/10.1016/j.margen.2019.100723 (2020).
Tipping, M. E. & Bishop, C. M. Probabilistic Principal Component Analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61, 611–622, doi:https://doi.org/10.1111/1467-9868.00196 (1999).
Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning. 2 edn, (Springer, 2008).
Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), <http://arxiv.org/abs/1511.07289> (2016).
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2818-2826.
Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization, <https://arxiv.org/abs/1412.6980> (2014).

No competing interests reported.

supplement.docx

Download PDF

Editorial decision: Major revision
30 Mar, 2022
Reviews received at journal
21 Mar, 2022
Reviewers agreed at journal
11 Mar, 2022
Reviewers invited by journal
11 Mar, 2022
Editor assigned by journal
11 Mar, 2022
Editor invited by journal
13 Feb, 2022
Submission checks completed at journal
13 Feb, 2022
First submitted to journal
04 Feb, 2022

You are reading this latest preprint version

Spectral Organ Fingerprints For Machine Learning-Based Intraoperative Tissue Classification With Hyperspectral Imaging

Status:

Version 1

Abstract

Figures

1 Background

2 Results

3 Discussion

4 Methods

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1