Machine Learning-derived Clinical Decision Algorithm for the Diagnosis of Hyperfunctioning Parathyroid Glands in Patients with Primary Hyperparathyroidism

doi:10.21203/rs.3.rs-2460003/v1

Download PDF

Research Article

Machine Learning-derived Clinical Decision Algorithm for the Diagnosis of Hyperfunctioning Parathyroid Glands in Patients with Primary Hyperparathyroidism

https://doi.org/10.21203/rs.3.rs-2460003/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Purpose

In patients with primary hyperparathyroidism (PHPT), planning for parathyroid surgery currently relies on the synthesis of clinical, laboratory, and imaging data by the clinician. Machine learning may assist in analyzing and integrating data to facilitate surgical decision making. To train and validate a Machine Learning-derived Clinical Decision Algorithm (_MLCDA) for the diagnosis of abnormal hyperfunctioning parathyroid glands using preoperative variables.

Methods

Four hundred and fifty-eight consecutive patients were evaluated from a single-institution retrospective dataset of PHPT patients who underwent combined 4D-CT and sestamibi SPECT/CT (MIBI) with subsequent parathyroidectomy from February 2013 to September 2016. Study cohort was divided into training (first 400 patients) and validation sets (remaining 58 patients). Sixteen preoperative clinical, laboratory, and imaging variables were evaluated. A random forest algorithm was programmed to select the best predictor variables and output a single clinical decision algorithm with the highest performance (_MLCDA). The _MLCDA was trained to predict the probability of a hyperfunctioning vs. normal gland for each of four parathyroid glands in a patient. Reference standard was 4-quadrant location on operative reports and pathological confirmation of adenoma or hyperplasia. Accuracy of _MLCDA was prospectively validated.

Results

Of 16 variables, the algorithm selected 3 variables for optimal prediction: combined 4D-CT and MIBI using 1) sensitive reading and 2) specific reading, and 3) crossproduct of serum calcium and parathyroid hormone levels, and outputted a _MLCDA using five probability categories for hyperfunctioning glands. The _MLCDA demonstrated excellent accuracy for correct classification in the training set (4D-CT + MIBI: 0.91 [95%CI 0.89–0.92]), and in the validation set (4D-CT + MIBI: 0.90 [95%CI 0.86–0.94], 4D-CT: 0.88 [95%CI 0.84–0.92], and MIBI: 0.88 [95%CI 0.84–0.92]).

Conclusion

Machine learning generated a clinical decision algorithm that accurately diagnosed hyperfunctioning parathyroid gland through classification into probability categories, which can be implemented for improved preoperative planning and convey diagnostic certainty.

Hyperparathyroidism

Machine Learning

Parathyroid

Sestamibi SPECT

Parathyroid 4D-CT

In patients with primary hyperparathyroidism (PHPT), preoperative planning for parathyroid surgery relies on thorough analysis and synthesis of a range of patient-specific data to guide the best surgical approach for each patient. Since the advent of minimally invasive parathyroidectomy (MIP), imaging localization has gained increasing importance by assisting surgeons in deciding between MIP and traditional bilateral neck exploration (BNE) (1). In general, patients with imaging findings of single gland disease undergo MIP, and patients with imaging findings of multigland disease or negative imaging undergo BNE (2). While accurate preoperative imaging is crucial to successful surgery, other clinical and laboratory variables provide valuable information that can be used to complement imaging data for improved surgical decision making.

The most widely used parathyroid preoperative imaging modalities are ultrasound, sestamibi (MIBI), and four-dimensional computed tomography (4D-CT), with most institutions using a combination of ultrasound and MIBI and performing 4D-CT as an adjunct modality (3, 4). However, in recent years, 4D-CT has emerged as an important imaging technique with high diagnostic accuracy in a variety of clinical scenarios, including more difficult to treat patients with multigland disease and mild PHPT (4–6). Studies comparing 4D-CT with MIBI and ultrasound have also demonstrated the superior accuracy of 4D-CT compared to the other two methods (4–9).

The most critical biological variables in PHPT are serum calcium and parathyroid hormone (PTH) concentrations, which are used for diagnosis and disease monitoring (10). PHPT presents in several different clinical settings ranging from asymptomatic PHPT, in which patients do not have overt symptomatology, to normocalcemic PHPT in which patients are discovered in the context of an evaluation of low bone mass, to the historically classic presentation of marked hypercalcemia, very high PTH levels and overt symptomatology (10). In general, higher calcium and PTH levels correlate with more severe disease, higher incidence of single gland disease (solitary adenomas), larger gland size, and conversely lower incidence of multigland, hyperplastic disease (11, 12). As a result, the serum calcium and PTH level has been utilized to predict probability of multigland disease in surgical (11, 12) and radiology studies (13, 14). The Wisconsin Index (WIN), defined as the product of preoperative serum calcium and PTH levels, is one proposed method for stratifying the biochemical profile (12).

Machine learning (ML), a subset of artificial intelligence (AI), is a data science method in which computers learn iteratively without being programmed with explicit rules (15). Since PHPT is a heterogeneous disease with a varying range of clinical and imaging features, ML may be implemented to develop individualized outcome prediction analytics for precision medicine. ML models have the potential to personalize health care through the analysis of a wide spectrum of patient data (15). For example, ML can learn to predict an individual patient’s outcome based upon the outcomes of other patients with similar features (15, 16). In ML, the algorithm is developed to predict the outcome by classifying patients in the training set and its accuracy is then evaluated in a validation set (16).

In this study, we proposed to train a ML algorithm to generate a clinical decision tree predicting localization of abnormal hyperfunctioning parathyroid glands (parathyroid adenoma or hyperplasia). To this end, we evaluated potential predictors using a wide range of patient-specific data from PHPT patients, including clinical, laboratory, and imaging variables. The primary endpoint of this study was to validate the performance of a decision tree derived from a ML random forest algorithm without human supervision to detect hyperfunctioning glands in a validation set, herein referred to as Machine Learning-derived Clinical Decision Algorithm (_MLCDA). Our goal was to harness the computational power of ML to develop a simple and parsimonious decision tree that can be easily used in routine clinical practice. We hypothesized that ML can accurately predict the probability of hyperfunctioning gland(s) in all 4 quadrants and convey the level of diagnostic confidence to the surgeon to improve preoperative planning.

Study design

A retrospective study was performed of consecutive patients with PHPT at our institution who underwent parathyroid 4D-CT and dual-phase MIBI single photon emission computed tomography/computed tomography (SPECT/CT) using a combined imaging protocol (4D-CT + MIBI) with subsequent parathyroidectomy. A prior study has reported the description, technical parameters of imaging acquisition, and diagnostic performance of 4D-CT + MIBI (9). We collected all relevant clinical, laboratory, and imaging data for predictor variables and operative and pathology reports for reference standard.

Figure 1 summarizes the study design. We trained ML algorithms to predict the probability of a parathyroid gland as a positive hyperfunctioning gland or negative normal gland, using a 4-quadrant, per-lesion analysis. All patients were assumed to have four parathyroid glands. The study population (n = 458 patients) was composed of a training set (n = 400 patients) and a validation set (n = 58 patients). The training set was composed of 400 patients (n = 522 hyperfunctioning glands/1,600 parathyroid glands) who underwent imaging from February 2013 to May 2016 and overlapped with patients from a prior study (9).

Following ML algorithm development on the training set, a validation set obtained from an additional retrospective review of 58 consecutive PHPT patients (n = 77 hyperfunctioning glands/232 parathyroid glands) who underwent imaging from May 2016 to September 2016 and subsequent parathyroidectomy. Thus, the clinical data in the validation set was locked up to the definition of the final signature in the training set. Patients with incomplete relevant clinical, biochemical or imaging datasets were excluded from the training and validation of the signature. Our study was HIPAA compliant and approved by the Institutional Review Board (IRB).

Candidate predictor variables

Candidate predictor variables were based on readily available preoperative variables. Four demographic variables were age, gender, race, and ethnicity. Three laboratory variables were preoperative serum calcium, serum PTH, and the product of preoperative serum calcium and PTH levels (Ca*PTH), also known as the aforementioned WIN. Nine imaging variables were 4D-CT, MIBI, 4D-CT + MIBI_sensitive, 4D-CT + MIBI_specific, number of abnormal glands on imaging, and four variables of parathyroid gland quadrant location on imaging (i.e. right upper quadrant). Since combining 4D-CT and MIBI imaging yields two results for each modality, we used two analytic approaches in the interpretation of the results, as described in a prior study (9). In the sensitive analysis (4D-CT + MIBI_sensitive), if either 4D-CT or MIBI was positive for a gland, then the result was considered positive. In the specific analysis (4D-CT + MIBI_specific), both 4D-CT and MIBI must be positive for the result to be considered positive. Three additional postoperative variables from pathology were included for the complex signature (discussed in Machine learning algorithms below): parathyroid weight, four parathyroid gland average weight, and four parathyroid total weight.

Reference standard

Location of abnormal parathyroid glands on operative reports and pathology reports of resected glands served as the reference standard for a normal gland (pathology negative) or abnormal hyperfunctioning gland (pathology positive). All patients underwent intraoperative PTH (IoPTH) monitoring and met the Miami Criterion, with a 50% or greater drop in IoPTH levels from the higher of the two pre-excision values (17). In this analysis, all unresected parathyroid glands were assumed to be normal glands (pathology negative) as all patients met Miami Criterion.

Clinical Decision Algorithm building

All preoperative clinical, laboratory, and imaging variables (n = 16) were evaluated as candidate predictor variables for clinical decision algorithm building, and three additional postoperative variables for parathyroid pathology weight were included for the second ML strategy. Using the training set, we developed a multivariable prediction model, i.e. the clinical decision algorithm, to diagnose hyperfunctioning parathyroid glands based on all predictor variables in the training set. Specifically, the signature outputs the probability that a parathyroid gland is a hyperfunctioning gland. Patients with a signature value > 0.5 were classified as positive for hyper functioning parathyroid gland, and negative otherwise. We have evaluated two ML strategies using random forest algorithms.

Our strategy was to generate the clinical decision algorithm using preoperative variables (n = 16). To this end, we programmed a Random Forest algorithm to generate a single, parsimonious, easy to use decision tree designed without human supervision. Our goal was to leverage the ML algorithm for use in routine clinical practice. The clinical decision algorithm is a single tree that is an approximation of the forest predictor. This tree is generated automatically by the random forest algorithm based upon the following criteria to reduce overfitting and to simplify for translation into clinical routine. First, a maximum combination of three informative candidate features was selected based on forward search and feature combination. Second, the maximal depth of the tree was predefined to be three. Third, classification and regression tree with pruning to consider the minimal tree connecting the root and leaves was used to combine features and generate the optimal candidate classification model, i.e. the signature, that achieved the best performance in terms of area under a receiver operating characteristic curve (AUC) estimated by three-fold cross-validation.

As an ancillary study, a “complex signature” using all pre- and postoperative variables (n = 19) was generated (eMethods 1 in the Supplemental Materials).

Validation of the clinical decision algorithm

The performance was tested in an external validation set of unseen data from a different cohort of patients than the training set. The training set was comparable to the validation set in terms of setting, eligibility criteria, outcome, and predictors.

Statistical analysis

Data were presented as frequency with percentages for qualitative variables and as mean with standard deviation for continuous variables. Calculated data with p values < 0.05 were considered as statistically significant. The performance of the machine learning algorithm for detecting hyperfunctioning parathyroid gland was calculated using a binary classifier system (normal vs. hyperfunctioning gland) and the accuracy was computed. All statistical analyses were performed using SPSS software 24.0 (IBM, Armonk, New York) and R (R Foundation for Statistical Computing, Vienna, Austria). AUC comparisons for correlated ROC curves were performed using Hanley and McNeil techniques.

Patient characteristics

Table 1 summarizes the patient demographics. The mean age of the patient cohort was 61 ± 14 years in the training set and 62 ± 12 in the validation set. 319 patients (79.8%) were female in the training set while they were 45 (77.6%) in the validation set. The median (interquartile range) calcium level was 10.7 mg/dL (10.4– 11.1) in the training set and 10.7 mg/dL (10.4–11.0) in the validation set, with normal ranges for serum calcium 8.4–10.2 mg/dL. PTH level was 91.3 pg/mL (67.9 − 121.0) in the training set and 91.1 pg/mL (69.0-137.7) in the validation set, with normal ranges for parathyroid hormone of 10–65 pg/mL. Patient demographic and biochemical data was similar between the two patient cohorts used for signature training and validation.

Table 1

**Patient demographics and baseline characteristics** Values represent mean ± SD (range). Parathyroid hormone: PTH. Normal ranges for parathyroid hormone:10–65 pg/mL and serum calcium: 8.4–10.2 mg/dL
	Training (n = 400)	Validation (n = 58)
Age (years): mean ± SD(range)	61 ± 14 (19–91)	62 ± 12 (28–79)
Females	62 ± 12 (19–88)	62 ± 12 (28–79)
Males	57 ± 17 (19–91)	64 ± 13 (36–77)
Sex (number of patients)
Females	319 (79.8%)	45 (77.6%)
Males	81 (20.2%)	23 (39.6%)
Ethnicity (number of patients)
White	275 (68.8%)	53 (91.3%)
Black	33 (8.3%)	2 (3.4%)
Hispanic	47 (11.8%)	2 (3.4%)
Asian	8 (2.0%)	1 (1.7%)
Others	37 (9.3%)	0 (0%)
Biochemical profile
PTH (pg/mL): median (IQR)	91.3 (67.9 − 121.0)	91.1 (69.0-137.7)
Calcium (mg/dL): median (IQR)	10.7 (10.4– 11.1)	10.7 (10.4, 11.0)
Number of patients (% of total)
Single Gland	321 (80.2%)	42 (72.4%)
Multigland	79 (19.8%)	16 (27.6%)

Algorithm building for Predicting Hyperfunctioning Gland

The _MLCDA included three variables and demonstrated excellent accuracy for correct classification in the training set of 400 patients: 0.91 [95% CI 0.89–0.92], and outperformed other clinical and imaging variables. Of sixteen preoperative variables, the three most important variables for prediction of hyperfunctioning gland were two imaging variables, 4D-CT + MIBI_sensitive and 4D-CT + MIBI_specific, and one laboratory variable, Ca*PTH. Figure 2 shows the _MLCDA, which is the most optimal single prediction tree developed from the training set, with respective probabilities of hyperfunctioning gland on pathology. The first branch of the prediction tree is whether a gland is positive on at least one of two imaging modalities (right branch) or negative on both imaging modalities (left branch). If either 4D-CT or MIBI or both modalities are positive, the probability of a hyperfunctioning gland increases to 97% when both modalities are positive, whereas the probability decreases to 75% if only one modality is positive. If both imaging modalities are negative, then the probability of a hyperfunctioning gland is dependent on Ca*PTH, a surrogate measure of biochemical profile, with increasing probability of hyperfunctioning gland with decreasing Ca*PTH. Clinically, this translates into increasing probability of false negative on imaging in milder biochemical profiles. The unsupervised _MLCDA stratified Ca*PTH into three categories: Ca*PTH > 1292, 676 < Ca*PTH < 1292, and Ca*PTH < 676, with corresponding increasing probabilities. There was no statistically significant incremental performance of more complex and deeper machine learning-derived signature presented in eResults 1 in Supplement.

Therefore, the _MLCDA identified five categories based on three prediction variables in the training set, with increasing probability of hyperfunctioning parathyroid gland: (i) Negative 4D-CT + MIBI_specific and Ca*PTH > 1292; (ii) Negative 4D-CT + MIBI_specific and 676 < Ca*PTH < 1292; (iii) Negative 4D-CT + MIBI_specific and Ca*PTH < 676; (iv) Positive 4D-CT + MIBI_sensitive (either 4D-CT or MIBI positive); (v) Positive 4D-CT + MIBI_specific (both 4D-CT and MIBI positive). The probability of a hyperfunctioning gland, based on percentage of positive lesions on pathology, for each category was respectively: (i) 3.8%; (ii) 8.0%; (iii) 19.4%; (iv) 75.0%; (v) 97.0%. Figure 3 shows clinical examples of how the _MLCDA can be used as a clinical decision tree and integrated into the radiology report to guide preoperative surgical planning.

Prospective validation of the algorithm for sestamibi SPECT/CT and 4D-CT

The _MLCDA was validated in the validation set of 58 patients (Fig. 4). The _MLCDA demonstrated excellent accuracy for correct classification in the validation set using 4D-CT + MIBI: 0.90 [95% CI 0.86–0.94]. The probability of a hyperfunctioning gland in the five categories was respectively: (i) 3.4%; (ii) 10.3%; (iii) 24.1%; (iv) 70.6%; (v) 96.0% and similar to the training set. Table 2 summarizes the diagnostic performance of the imaging modalities in the training and validation dataset using a 4-quadrant analysis, with similar diagnostic performance in training and validation sets.

Table 2

Summary of Diagnostic Performance of the Imaging Modalities in the Training and Validation Sets
	Training set	Validation set
	4D-CT + MIBI	4D-CT + MIBI	4D-CT alone	MIBI alone
TP	420	60	57	51
FP	50	7	20	26
TN	1028	148	148	153
FN	102	17	7	2
Sensitivity	0.8 (0.77–0.84)	0.78 (0.69–0.87)	0.89 (0.81–0.97)	0.96 (0.91–1.01)
Specificity	0.95 (0.94–0.97)	0.95 (0.92–0.99)	0.88 (0.83–0.93)	0.85 (0.8–0.91)
Likelihood Ratio +	17.35 (13.19–22.81)	17.25 (8.29–35.93)	7.48 (4.91–11.39)	6.62 (4.62–9.49)
Likelihood Ratio -	0.2 (0.17–0.24)	0.23 (0.15–0.35)	0.12 (0.06–0.25)	0.04 (0.01–0.17)
False positive rate	0.05 (0.03–0.06)	0.05 (0.01–0.08)	0.12 (0.07–0.17)	0.15 (0.09–0.2)
False negative rate	0.2 (0.16–0.22)	0.22 (0.13–0.28)	0.11 (0.03–0.16)	0.04 (-0.01-0.07)
Prob of disease	0.33 (0.3–0.35)	0.33 (0.27–0.39)	0.28 (0.22–0.33)	0.23 (0.17–0.28)
Pred value positive	0.89 (0.87–0.92)	0.9 (0.82–0.97)	0.74 (0.64–0.84)	0.66 (0.56–0.77)
p(pos test wrong)	0.11 (0.08–0.13)	0.1 (0.03–0.18)	0.26 (0.16–0.36)	0.34 (0.23–0.44)
Pred value negative	0.91 (0.89–0.93)	0.9 (0.85–0.94)	0.95 (0.92–0.99)	0.99 (0.97-1)
p(neg test wrong)	0.09 (0.07–0.11)	0.1 (0.06–0.15)	0.05 (0.01–0.08)	0.01 (0-0.03)
p(test positive)	0.29 (0.27–0.32)	0.29 (0.23–0.35)	0.33 (0.27–0.39)	0.33 (0.27–0.39)
p(test negative)	0.71 (0.68–0.73)	0.71 (0.65–0.77)	0.67 (0.61–0.73)	0.67 (0.61–0.73)
Overall accuracy	0.91 (0.89–0.92)	0.9 (0.86–0.94)	0.88 (0.84–0.92)	0.88 (0.84–0.92)
The Machine Learning-derived Clinical Decision Algorithm (_MLCDA) was trained in the training set for hybrid imaging with 4D-CT + MIBI and was then validated using the validation set. As an ancillary study, we applied this same _MLCDA to 4D-CT only and MIBI only, whereas in the original _MLCDA, imaging is positive based upon the analysis of two imaging modalities (4D-CT + MIBI).

Prospective validation of the algorithm for sestamibi SPECT/CT alone and 4D-CT alone

Since both 4D-CT and MIBI may not be widely available or routinely performed, we also evaluated the predictive accuracy of the signature if only one imaging modality is used using the validation set. Similar results were observed when using only one modality (Fig. 4), except that there were four probability categories since no patients had two positive imaging studies. The _MLCDA demonstrated excellent accuracy in the validation set for 4D-CT alone: 0.88 (95% CI 0.84–0.92) as well as MIBI alone: 0.88 95% CI 0.84–0.92). For 4D-CT, the probability of a hyperfunctioning gland in the four probability categories was respectively: (i) 8.2%; (ii) 10.3%; (iii) 24.1%; (iv) 89.1%, and for MIBI: (i) 6.5%; (ii) 12.3%; (iii) 33.3%; (iv) 96.2%. Compared to MIBI, 4D-CT had a lower probability of hyperfunctioning gland when imaging was positive but had an overall lower probability of false negative imaging when imaging was negative. Figure 5 summarizes the probability of a hyperfunctioning gland in the five categories in both training and validation sets.

Using machine learning, we developed and validated a simple clinical decision algorithm to diagnose hyperfunctioning parathyroid gland based on three readily available imaging and laboratory variables. Performance of the algorithm for correct binary classification of a hyperfunctioning gland was excellent and outperformed all other variables. Our simple decision tree can be easily implemented into clinical practice to provide an evidence-based probability map of hyperfunctioning and normal glands based on imaging and biochemical profile to guide the surgeon in individualized preoperative planning.

Currently, the paradigm for radiology reports is based on descriptive and subjective verbiage (i.e. “most likely” or “consistent with”) and typically does not convey a degree of diagnostic confidence or probability of disease(18). This leaves the onus upon the clinician to interpret the verbiage of the report, in addition to integrating other variables and data, to formulate a diagnostic or treatment decision(18, 19). Additionally, 4D-CT and MIBI imaging are generally interpreted as a binary result (positive or negative), however patients with PHPT exist on a spectrum of disease and their imaging and biochemical data should accurately reflect that spectrum for optimal patient care. To address this problem, we developed a clinical decision tree using ML that is highly accurate, evidence-based, and defines five distinct probability categories for a hyperfunctioning gland based on serum calcium and PTH levels and 4D-CT and MIBI imaging, with respective probabilities: (i) 3.8%; (ii) 8.0%; (iii) 19.4%; (14) 75.0%; (v) 97.0%. Using a separate dataset, the results from the training set were validated with comparable probabilities. The decision tree can be used to guide clinical decision making to decide the type of parathyroid surgery (MIP vs. BNE). For instance, in a patient with 96% probability for one gland and 3% probability for three glands, a MIP appears to be the best option. On the contrary, in a patient with 24% probability for all four glands, a BNE can be planned with greater confidence.

Using 16 different input variables, the ML signature selected 3 variables for optimal prediction of a hyperfunctioning gland and discarded 13 other variables. The signature also demonstrated that in glands with any positive imaging, no other type of variable was needed for optimal prediction besides imaging, however in glands with negative imaging, the biochemical profile (Ca*PTH) was the most important determinant of probability of a hyperfunctioning gland or conversely, of a normal gland and false negative on imaging. Interestingly, the Ca*PTH cutoff thresholds automatically derived by the ML signature were similar to the original Wisconsin Index (WIN) risk categories proposed by Mazeh et al. (12), with Ca*PTH < 676, 676 < Ca*PTH < 1292, and Ca*PTH > 1292 for our ML algorithm compared to Ca*PTH < 800, 801 < Ca*PTH < 1600, and Ca*PTH > 1600 for WIN. This concordance in Ca*PTH cutoff thresholds derived by separate methodologies suggests that biochemical profile assessed using Ca*PTH does reflect underlying PHPT disease biology and severity, with more severe biochemical profile correlating with increasing probability of single gland disease. Indeed, WIN is a component in the composite multigland disease score proposed by Sepahdari et al. (13), a prospectively validated predictive scoring system for likelihood of MGD based on 4D-CT findings and WIN (13, 14). However, Edafe et al. demonstrated that WIN did not accurately differentiate between single and multigland disease in a large cohort of 624 patients (20). In contrast to WIN, our Ca*PTH cutoffs were derived through ML without supervision, and our prediction tree incorporates imaging criteria of 4D-CT and MIBI and may be further adapted using ML to potentially predict multigland disease.

Our _MLCDA outperformed all individual variables, including imaging variables, such as combined 4D-CT + MIBI_sensitive, 4D-CT alone, and MIBI alone, which had the highest performances of single variables. A prior study had compared 4D-CT vs. MIBI in the same cohort of patients and showed that 4D-CT was superior to MIBI and combined 4D-CT + MIBI does not provide added benefit over 4D-CT alone suggesting the limited value of MIBI (9). Contrary to that study, the ML signature selected 4D-CT + MIBI_specific as one of three important variables to include in the predictive tree model. When both imaging modalities were positive (positive 4D-CT + MIBI_specific), the probability for a hyperfunctioning gland increases to 97% from 75% when only one modality was positive. This suggests that MIBI imaging does provide incremental clinical value by increasing diagnostic confidence, which may not be readily detected using standard diagnostic accuracy measures and requires the power of a ML algorithm to work through all possible permutations. The conclusions of the two studies may be partly explained by the fact that MIBI had a slightly higher specificity compared to 4D-CT in our prior study (99% vs. 96%, respectively) and PPV (96% vs. 90%, respectively), probably due to lower false positives of MIBI(9), and these subtle differences were determined important for prediction by the ML random forest algorithm.

Despite the growing research and applications of ML and AI in medical imaging, only two prior studies have evaluated the use of ML in hyperparathyroidism and parathyroid diseases (21, 22). Imbus et al. used a ML algorithm to differentiate between single and multigland disease (21) and Somnay et al. evaluated ML to diagnose patients with PHPT (22). Both studies used demographic, clinical, and laboratory variables as predictors, but neither study included imaging variables. While both studies showed high accuracy of ML, the algorithms were not translated into a readily implemented clinical algorithm unlike in our study. Similarly, our complex signature provided a deeper understanding of the relative importance and relationships of different variables in prediction than our simple _MLCDA, however the complex signature could not be translated into clinical practice unless implemented on a software and also requires postoperative variables. Therefore, we pursued prospective validation of the simple _MLCDA. Our study differs from prior studies as our ML signature simplified a dataset of many variables into a clinical decision tree using only three variables that can be implemented into radiology reports to communicate probabilities to the surgeon for surgical planning.

Another originality of our work is that we implemented several strategies to limit the risk of false-positive findings and optimized simplicity of the model for translation in clinical practice. First, we use a large sample size for algorithm development and training. Second, we decided to select an optimal decision tree rather than using thousands of trees from the ML algorithm. The decision tree was optimized in the training set to predict pathology as a binary variable. Thus, the output was five classes redefining probability of disease as five categories corresponding a spectrum from lowest to highest probability of a hyperfunctioning gland. Third, we created a low-dimensional model. Of note, we evaluated 16 candidate variables, but the algorithm demonstrated that most of the variables were highly correlated and therefore redundant without providing added diagnostic value. The unsupervised random forest algorithm demonstrated in the training set that only three variables were necessary: one continuous variable (Ca*PTH) and two binary variables (4D-CT+/- and MIBI+/-). Using more than three variables did not significantly improve the performance of the model in our training set. In practice, this means that the number of dimensions in this signature is limited and that the signature is likely underfitted, as larger datasets would allow for developing more complex models achieving higher performance. Fourth, this low-dimensional signature was tested on a validation set that was not used for training and reached similar performance. Most importantly, unlike some AI/ML algorithms, our ML-derived clinical decision tree is not a black box and is a ready to use clinical tool that can be implemented across institutions to replace radiology verbiage with evidence-based probabilities. This spectrum of probability with different degree of certainty could lead to a precision medicine approach and increase diagnostic confidence.

There are several limitations of our study. First, our cohort was acquired from a single institution, which limits the generalizability of our findings to other institutions and practice settings. Second, our algorithm is based on using combined 4D-CT and MIBI, which may not be available or routinely performed at other institutions, however we tried to address this limitation by adapting the prediction algorithm for 4D-CT or MIBI alone. This resulted in slightly lower, but still high prediction probabilities when imaging was positive: 89% for 4D-CT and 96% for MIBI. While 4D-CT had a lower probability of hyperfunctioning gland when positive compared to MIBI, it had an overall lower probability of false negative imaging when imaging was negative (24.1% vs. 33.3%), particularly when Ca*PTH < 676, which partly explains the superior performance of 4D-CT in multigland disease. A third limitation is the potential overfitting for the model to the training dataset. We reduced overfitting by specifying a maximum combination of three informative candidate variables through forward search and feature combination. Lastly, we did not include parathyroid gland size as a variable in our simple signature because our goal was to create a prediction based only on preoperative variables, although the complex signature demonstrated parathyroid weight on pathology was the most important predictor variable. We hope to address this limitation in a future study by using estimated parathyroid weight on 4D-CT.

A ML-derived clinical decision algorithm can accurately predict hyperfunctioning parathyroid glands using only two imaging and one laboratory variables. The simplified prediction tree derived from ML can be used to convey degree of diagnostic confidence to the surgeon to improve preoperative planning. Future studies include using ML to adapt the prediction tree to predict probability of single and multigland disease and recommend the correct type of surgery for individual patients for precision medicine.

4D = four-dimensional

CT = computed tomography

MIBI = sestamibi

SPECT = single photon emission computed tomography

MIP = minimally invasive parathyroidectomy

ioPTH = intraoperative parathyroid hormone

PHTP = primary hyperparathyroidism

Acknowledgements

We thank Dr. Elias Kazam for his contributions and expertise in helping us develop the combined 4D-CT and sestamibi SPECT/CT imaging protocol.

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Author Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Randy Yeh, Laurent Dercle, Jennifer Kuo and Bernice Huang. The first draft of the manuscript was written by Randy Yeh and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Ethics approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Institutional Review Board of Columbia University Medical Center/New York Presbyterian Hospital.

Consent to participate

Informed consent was waived by the Institutional Review Board for this retrospective study.

Consent to publish

Not applicable.

Disclosures

None

Rodgers SE, Hunter GJ, Hamberg LM, Schellingerhout D, Doherty DB, Ayers GD, et al. Improved preoperative planning for directed parathyroidectomy with 4-dimensional computed tomography. Surgery. 2006;140(6):932-40; discussion 40-1. Epub 2006/12/26. doi: 10.1016/j.surg.2006.07.028. PubMed PMID: 17188140.
Hoang JK, Sung WK, Bahl M, Phillips CD. How to perform parathyroid 4D CT: tips and traps for technique and interpretation. Radiology. 2014;270(1):15-24. Epub 2013/12/21. doi: 10.1148/radiol.13122661. PubMed PMID: 24354373.
Greene AB, Butler RS, McIntyre S, Barbosa GF, Mitchell J, Berber E, et al. National trends in parathyroid surgery from 1998 to 2008: a decade of change. J Am Coll Surg. 2009;209(3):332-43. Epub 2009/09/01. doi: 10.1016/j.jamcollsurg.2009.05.029. PubMed PMID: 19717037.
Kelly HR, Hamberg LM, Hunter GJ. 4D-CT for preoperative localization of abnormal parathyroid glands in patients with hyperparathyroidism: accuracy and ability to stratify patients by unilateral versus bilateral disease in surgery-naive and re-exploration patients. AJNR American journal of neuroradiology. 2014;35(1):176-81. doi: 10.3174/ajnr.A3615. PubMed PMID: 23868155.
Day KM, Elsayed M, Beland MD, Monchik JM. The utility of 4-dimensional computed tomography for preoperative localization of primary hyperparathyroidism in patients not localized by sestamibi or ultrasonography. Surgery. 2015;157(3):534-9. doi: 10.1016/j.surg.2014.11.010. PubMed PMID: 25660183.
Eichhorn-Wharry LI, Carlin AM, Talpos GB. Mild hypercalcemia: an indication to select 4-dimensional computed tomography scan for preoperative localization of parathyroid adenomas. Am J Surg. 2011;201(3):334-8; discussion 8. doi: 10.1016/j.amjsurg.2010.08.033. PubMed PMID: 21367374.
Kukar M, Platz TA, Schaffner TJ, Elmarzouky R, Groman A, Kumar S, et al. The use of modified four-dimensional computed tomography in patients with primary hyperparathyroidism: an argument for the abandonment of routine sestamibi single-positron emission computed tomography (SPECT). Ann Surg Oncol. 2015;22(1):139-45. doi: 10.1245/s10434-014-3940-y. PubMed PMID: 25074663.
Starker LF, Mahajan A, Bjorklund P, Sze G, Udelsman R, Carling T. 4D parathyroid CT as the initial localization study for patients with de novo primary hyperparathyroidism. Ann Surg Oncol. 2011;18(6):1723-8. doi: 10.1245/s10434-010-1507-0. PubMed PMID: 21184187.
Yeh R, Tay YD, Tabacco G, Dercle L, Kuo JH, Bandeira L, et al. Diagnostic Performance of 4D CT and Sestamibi SPECT/CT in Localizing Parathyroid Adenomas in Primary Hyperparathyroidism. Radiology. 2019;291(2):469-76. Epub 2019/03/06. doi: 10.1148/radiol.2019182122. PubMed PMID: 30835187.
Bilezikian JP. Primary Hyperparathyroidism. J Clin Endocrinol Metab. 2018;103(11):3993-4004. Epub 2018/07/31. doi: 10.1210/jc.2018-01225. PubMed PMID: 30060226; PubMed Central PMCID: PMCPMC6182311.
Lim JY, Herman MC, Bubis L, Epelboym I, Allendorf JD, Chabot JA, et al. Differences in single gland and multigland disease are seen in low biochemical profile primary hyperparathyroidism. Surgery. 2017;161(1):70-7. Epub 2016/11/17. doi: 10.1016/j.surg.2016.08.054. PubMed PMID: 27847113.
Mazeh H, Chen H, Leverson G, Sippel RS. Creation of a "Wisconsin index" nomogram to predict the likelihood of additional hyperfunctioning parathyroid glands during parathyroidectomy. Annals of surgery. 2013;257(1):138-41. Epub 2012/07/18. doi: 10.1097/SLA.0b013e31825ffbe1. PubMed PMID: 22801087.
Sepahdari AR, Bahl M, Harari A, Kim HJ, Yeh MW, Hoang JK. Predictors of Multigland Disease in Primary Hyperparathyroidism: A Scoring System with 4D-CT Imaging and Biochemical Markers. AJNR American journal of neuroradiology. 2015;36(5):987-92. Epub 2015/01/04. doi: 10.3174/ajnr.A4213. PubMed PMID: 25556203.
Sho S, Yilma M, Yeh MW, Livhits M, Wu JX, Hoang JK, et al. Prospective Validation of Two 4D-CT-Based Scoring Systems for Prediction of Multigland Disease in Primary Hyperparathyroidism. AJNR American journal of neuroradiology. 2016;37(12):2323-7. Epub 2016/09/24. doi: 10.3174/ajnr.A4948. PubMed PMID: 27659191.
Choy G, Khalilzadeh O, Michalski M, Do S, Samir AE, Pianykh OS, et al. Current Applications and Future Impact of Machine Learning in Radiology. Radiology. 2018;288(2):318-28. Epub 2018/06/27. doi: 10.1148/radiol.2018171820. PubMed PMID: 29944078; PubMed Central PMCID: PMCPMC6542626.
Kohli M, Prevedello LM, Filice RW, Geis JR. Implementing Machine Learning in Radiology Practice and Research. AJR American journal of roentgenology. 2017;208(4):754-60. Epub 2017/01/27. doi: 10.2214/ajr.16.17224. PubMed PMID: 28125274.
Carneiro DM, Solorzano CC, Nader MC, Ramirez M, Irvin GL, 3rd. Comparison of intraoperative iPTH assay (QPTH) criteria in guiding parathyroidectomy: which criterion is the most accurate? Surgery. 2003;134(6):973-9; discussion 9-81. Epub 2003/12/12. doi: 10.1016/j.surg.2003.06.001. PubMed PMID: 14668730.
Panicek DM, Hricak H. How Sure Are You, Doctor? A Standardized Lexicon to Describe the Radiologist's Level of Certainty. AJR American journal of roentgenology. 2016;207(1):2-3. Epub 2016/04/12. doi: 10.2214/ajr.15.15895. PubMed PMID: 27065212.
Schwartz LH, Panicek DM, Berk AR, Li Y, Hricak H. Improving communication of diagnostic radiology findings through structured reporting. Radiology. 2011;260(1):174-81. Epub 2011/04/27. doi: 10.1148/radiol.11101913. PubMed PMID: 21518775; PubMed Central PMCID: PMCPMC3121011.
Edafe O, Collins EE, Ubhi CS, Balasubramanian SP. Current predictive models do not accurately differentiate between single and multi gland disease in primary hyperparathyroidism: a retrospective cohort study of two endocrine surgery units. Annals of the Royal College of Surgeons of England. 2018;100(2):140-5. Epub 2017/10/13. doi: 10.1308/rcsann.2017.0112. PubMed PMID: 29022783; PubMed Central PMCID: PMCPMC5838681.
Imbus JR, Randle RW, Pitt SC, Sippel RS, Schneider DF. Machine learning to identify multigland disease in primary hyperparathyroidism. The Journal of surgical research. 2017;219:173-9. Epub 2017/10/29. doi: 10.1016/j.jss.2017.05.117. PubMed PMID: 29078878; PubMed Central PMCID: PMCPMC5661967.
Somnay YR, Craven M, McCoy KL, Carty SE, Wang TS, Greenberg CC, et al. Improving diagnostic recognition of primary hyperparathyroidism with machine learning. Surgery. 2017;161(4):1113-21. Epub 2016/12/19. doi: 10.1016/j.surg.2016.09.044. PubMed PMID: 27989606; PubMed Central PMCID: PMCPMC5367958.

Supplementalfigures.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Machine Learning-derived Clinical Decision Algorithm for the Diagnosis of Hyperfunctioning Parathyroid Glands in Patients with Primary Hyperparathyroidism

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusion

Figures

Introduction

Materials And Methods

Study design

Candidate predictor variables

Reference standard

Clinical Decision Algorithm building

Validation of the clinical decision algorithm

Statistical analysis

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1