Design, Implementation, and Evaluation of the Computer-aided Clinical Decision Support System based on Learning-to-Rank: Collaboration between physicians and machine learning in the differential diagnosis process

doi:10.21203/rs.3.rs-2068201/v2

Background

We are researching, developing, and publishing the Clinical Decision Support System based on Learning-to-Rank. The main objectives are 1) Supporting differential diagnoses by internists and general practitioners and 2) Preventing diagnostic errors by physicians. The main features are that "A physician inputs a patient's symptoms, findings, and test results to the system, and the system outputs a ranking list of possible diseases."

Method

The software libraries for machine learning and artificial intelligence are TensorFlow and TensorFlow Ranking. The prediction algorithm is Learning-to-Rank., with a listwise approach. The ranking metric is Normalized Discounted Cumulative Gain (NDCG). The loss functions are Approximate NDCG (A-NDCG) and Gumbel Approximate NDCG (G-A-NDCG). We evaluated the machine learning performance on k-fold cross-validation. We evaluated the differential diagnosis performance with actual cases.

Results

The machine learning performance of our system was much higher than that of the conventional system. The machine learning performance using G-A-NDCG was slightly higher than that of A-NDCG. The differential diagnosis performance of our system was much higher than that of the conventional system. We have shown that the Clinical Decision Support System prevents physicians' diagnostic errors due to confirmation bias.

Conclusions

We have demonstrated that the Clinical Decision Support System is useful for supporting differential diagnoses and preventing diagnostic errors. We believe that Differential diagnosis by physicians and Learning-to-Rank by machine has a high affinity. We found that Information Retrieval and Clinical Decision Support Systems have much in common (Target data, Learning-to-Rank, etc.). We believe that Clinical Decision Support Systems have the potential to support 1) recall of rare diseases, 2) differential diagnoses for difficult-to-diagnose diseases, and 3) prevention of diagnostic errors. We also believe that our system has the potential to evolve into an Explainable Clinical Decision Support System.

Clinical decision support system

Diagnosis decision support system

Differential diagnosis

Diagnostic error

Information Retrieval

Learning to rank

Listwise approach

Approximate NDCG

Gumbel approximate NDCG

Explainable Artificial Intelligence

We are researching, developing, and publishing the Clinical Decision Support System (CDSS) based on Learning-to-Rank (LTR). [1], [2]

This paper discusses our system's design, implementation, and evaluation.

1.1 Diagnostic errors and Clinical Decision Support System

Medical errors are among the most critical safety issues in today's healthcare. Medical errors cause the most significant damage (human and economic) to the public.

The well-known report "To Err Is Human." reports that 44,000 to 98,000 patients die annually in the United States due to medical errors. Deaths due to medical errors are more incredible than deaths due to the three leading causes of death (automobile accidents, breast cancer, and AIDS). [3]

Diagnostic errors are a type of medical error.

Briefly, diagnostic errors are as follows:

A delayed diagnosis
A wrong diagnosis
A missed diagnosis [4]

We believe the CDSS will be a competent partner with physicians to prevent diagnostic errors.

In clinical practice, internists and general practitioners also want the practical application of CDSS. [5]

1.2 Main objectives of the Clinical Decision Support System

This paper's main objectives of the Clinical Decision Support System (CDSS) are as follow:

Supporting differential diagnoses by internists and general practitioners
Preventing diagnostic errors by physicians

1.3 Main features of the Clinical Decision Support System

This paper's main features of the Clinical Decision Support System (CDSS) are as follow:

A physician inputs a patient’s symptoms, findings, and test results to the system, and the system outputs a ranking list of possible diseases.

The input information is as follows:

Subjective symptoms
Objective findings
Physical findings
Laboratory test results
Imaging test results
Other Information

(From now on, referred to as "inputted symptoms")

The output information is as follows:

A ranking list of possible diseases

(From now on, referred to as "predicted diseases")

Clinical Decision Support system (CDSS) for Differential Diagnosis (DDx) is also known as Diagnostic Decision Support System (DDSS). [6]

1.4 Example of the Clinical Decision Support System

Figure 1 shows the Example of the prediction screen of our system.

Table 1 shows the Example of the predicted results of our system.

On the Internet, our system is open to healthcare professionals.

1.5 Figures and Tables

Figure 1 Example of the prediction screen of our system

For details, see: 7.2.2. Difficult-to-diagnose disease with few characteristic symptoms

Table 1 Example of the predicted results of our system

	Inputted symptoms			Score	Predicted diseases
1	Fever		1	1.61	Acute HIV-1 infection
2	Headache		2	1.51	Polyneuropathy
3	Sore throat		3	0.91	Acute viral meningitis
4	Consciousness indistinctness		4	0.88	West Nile fever
5	Chills		5	0.77	Cat-scratch disease
6	Muscles ache		6	0.46	Acute Q fever
7	Swallowing pain	→	7	0.23	Hepatitis A
8	Pharyngolaryngeal abnormality		8	0.21	Chronic fatigue syndrome
9	Aphasia		9	0.13	Sepsis
10	Apraxia		10	0.12	Toxoplasmosis
11	Fatigue				...
12	Muscle weakness
13	Anorexia
14	Weight loss
15	Dementia

For details, see: 7.2.2. Difficult-to-diagnose disease with few characteristic symptoms

2.1. Differential Diagnosis process by physicians and Learning-to-Rank by machines

The Differential Diagnosis (DDx) process by experienced physicians is an iterative process with the following steps:

1) Perform medical examinations to obtain information about the diseases.

2) Recall multiple differential diseases.

3) Refine the recalled differential diseases.

4) Rank the refined differential diseases. [7]

Learning-to-Rank (LTR) is one of the Machine Learning (ML) techniques. LTR is used to construct ranking models for Information Retrieval (IR) systems. [8]

We believe that the DDx process by experienced physicians is highly affinitive to LTR by machines.

LTR includes the following approaches:

- Pointwise approach

- Pairwise approach

- Listwise approach [8]

From the perspective of LTR, the DDx process by experienced physicians IS NOT a pointwise or pairwise approach.

- Pointwise approach:

- Score one differential disease at a time.

- Pairwise approach:

- Compare two differential diseases at a time.

This process IS a listwise approach.

- Listwise approach:

1) Recall multiple differential diseases

2) Refine the recalled differential diseases

3) Rank the refined differential diseases

Once again, we believe the DDx process is highly affinitive to LTR, especially the listwise approach.

2.2. Case data for Clinical Decision Support System

The case data (= training data) for CDSS is prepared using a literature base. [9]

Real World Data (RWD) has not been validated its reliability.

We do not use them as case data for CDSS.

The medical literature includes the following types:

- Medical textbooks

- Medical treatises

- Medical articles

- Case reports

(From now on, referred to as "literature")

Good literature, such as case reports, contains information on confirmed disease(s) and (multiple) differential diseases.

Excellent literature, such as Clinical Problem Solving (CPS), contains information on confirmed disease(s) and (multiple and changing) differential diseases by following the DDx process by experienced physicians. [10]

The information discussed in case reports is as follows:

- Symptoms

- Confirmed disease(s)

- Differential diseases (related or to be excluded)

The procedure for making the case data for CDSS is as follows:

1) Select the literature

2) Retrieve the information on cases by text-mining from the literature

3) Convert the retrieved data by text-mining to the metadata

4) Store the metadata in the database

Technologies have already been developed to automatically text-mining information on the only confirmed disease from the abstracts of case reports. [9]

No technology has yet been developed to automatically text-mining information on confirmed disease(s) and (multiple) differential diseases from the body of literature.

No technology has yet been developed to convert retrieved information by text-mining to metadata automatically.

2.3. Information Retrieval and Clinical Decision Support System

Information Retrieval (IR) is a technique for retrieving information from information resources that match objectives. [8]

Google Scholar is a primary IR service that targets scholarly literature on the Internet.

IR (Google Scholar, etc.) and CDSS have much in common (target data, metadata, algorithm, etc.).

Table 2 shows the Information Retrieval and Clinical Decision Support System.

Retrieval algorithms for IR often use LTR, especially the listwise approach; we believe CDSS should use several IR technologies (LTR, etc.).

2.4. Conventional Clinical Decision Support Systems

Various leading researchers have reported on CDSS based on ML. [11], [12.], [13], [14], and [15]

The output of these systems is "predicted diseases." It is "a ranking list of possible diseases." Therefore, these systems are also a type of CDSS based on LTR. However, we assume that the prediction algorithm of these systems uses the pointwise approach. In addition, we assume that the case data of these systems use only confirmed disease information.

We believe that these systems have the following problems:

- The predictive algorithms are LTR with a pointwise approach.

- These algorithms are less affinitive to the DDx process by experienced physicians.

- The case data does not include information on differential diseases.

- These algorithms do not use the relationship between confirmed disease(s) and differential diseases.

2.5. Figures and Tables

Table 2

Information Retrieval and Clinical Decision Support System
Items	Information Retrieval	Clinical Decision Support System
Target data	Medical literature	←
Method of retrieving target data	Web crawlers, etc.	Selection by physicians
Metadata	Symptoms, Diseases	←
Target data → Metadata	Search engine, etc.	Physicians
Algorithm	Learning-to-Rank	←
Input data	Symptoms, Diseases	Symptoms
Output data	Ranking list of useful medical literature	Ranking list of possible diseases
Evaluation method	Subjective evaluation Physicians	Objective evaluation Case reports Evaluation Functions

3.1. Design principles

To address the issues of conventional CDSS, the design principles of our system are as follows:

- The prediction algorithms should be higher affinitive to the DDx process by experienced physicians.

- The case data should include not only information on confirmed disease(s) but also information on differential diseases.

- These algorithms should utilize the relationship between confirmed disease(s) and differential diseases.

- To focus on commonalities between IR and CDSS, utilize various IR technologies for CDSS.

3.2. Library for Learning-to-Rank

We used TensorFlow and TensorFlow Ranking as our system's Machine Learning (ML) libraries to satisfy the design principles.

TensorFlow Ranking is a library for Learning-to-Rank (LTR). The main targets for TensorFlow Ranking are Information Retrieval (IR) systems and Recommendation systems. [16], [17]

For the ranking metrics of LTR, we selected Normalized Discounted Cumulative Gain (NDCG). NDCG is the ranking metric of LTR (listwise approach). [8]

As discussed before, we believe that the calculation algorithm of NDCG is more affinitive to the DDx process by experienced physicians.

For the loss function of LTR, we selected Approximate NDCG loss and Gumbel Approximate NDCG loss.

Approximate NDCG loss is an approximation for NDCG. It is a differentiable approximation based on the logistic function. [18]

Gumbel Approximate NDCG loss is the same as the Approximate NDCG loss. Its logits are sampled from the Gumbel distribution. [19]

3.3. Case date for Learning-to-Rank with the listwise approach

The case data for conventional CDSS based on LTR (pointwise approach) has the following information:

- Symptoms

- Confirmed disease

Table 3 shows the Example of case data (pointwise approach).

These have only information on a confirmed disease.

As discussed before, technologies have already been developed to automatically text-mining this information from the abstracts of case reports.

The case data for our CDSS based on LTR (listwise approach) has the following information:

- Symptoms

- Confirmed disease(s) and these scores

- Differential diseases (related or to be excluded) and these scores

Table 4 shows the Example of case data (listwise approach).

This information has not only confirmed disease(s) but also differential diseases. In addition, these diseases are assigned a score according to possibility. This information is described not only in the abstracts of literature but also in the bodies.

Thus, the Information Retrieval (IR) system should parse the abstracts and the bodies.

3.4. Figures and Tables

Table 3

Example of case data (pointwise approach)
	Code	Observed symptoms		Code	Diseases
1	fever	Fever		548	Acute HIV-1 infection
2	head	Headache
3	sore	Sore throat
4	myalg	Muscles ache
5	fatig	Fatigue	→
6	weigh	Weight loss
7	arthralg	Arthralgia
8	diarrh	Diarrhea
9	lymphn	Lymphadenopathy
10	mening	Meningitis
	…
Based on: case data of our system

Table 4

Example of case data (listwise approach)
	Code	Observed symptoms		Scores	Code	Diseases
1	fever	Fever		17.078	548	Acute HIV-1 infection
2	head	Headache		12.086	296	Acute hepatitis
3	sore	Sore throat		11.250	102	Toxoplasmosis
4	myalg	Muscles ache		11.000	491	Severe fever with thrombocytopenia syndrome (SFTS)
5	fatig	Fatigue	→	10.836	391	Osteomyelitis
6	weigh	Weight loss		10.836	589	Polyneuropathy
7	arthralg	Arthralgia		10.836	641	Coccidioidomycosis
8	diarrh	Diarrhea		10.664	627	Cat-scratch disease
9	lymphn	Lymphadenopathy		10.500	541	Infectious endocarditis
10	mening	Meningitis		10.414	989	Dengue (hemorrhagic) fever
	…					...
Citation: case data of our system

4.1. Database

4.1.1. Overview

The contents of the database are as follows:

- Symptoms master data

- Diseases master data

- Case data

To improve the predictive performance of the CDSS, we believe it is necessary to define strict criteria for symptoms, diseases, and cases.

The criteria we defined for target cases are as follows:

Difficult-to-diagnose cases that internists and general practitioners may close encounter in actual clinical practice

The citations of the cases are from the literature collected by us.

4.1.2. Symptom master data

The symptom master data is used as "inputted symptoms."

Information in the symptom master data is as follows:

- Subjective symptoms

- Objective findings

- Physical findings

- Laboratory test results

- Imaging tests results

- Other Information

We used a qualitative expression, not a numerical one, for quantitative data.

We used an imaging finding, not imaging data, for imaging test results.

We did not use quantity or quality, except for special findings (ex: masses, atelectasis, pleural effusion, pneumothorax, etc.).

Table 5 shows the Example of expression in the symptom master data.

One code defines multiple similar symptoms. The reason is to support the input of symptoms by physicians.

The code system and the name of the symptoms in the symptom master data are based on our specifications.

The number of codes in the symptom master data is 583.

The total number of symptoms is approximately 1,200.

Table 6 shows the Example of the symptom master data.

4.1.3. Disease master data

The disease master data is used as "predicted diseases."

One record includes confirmed disease(s) and differential diseases (related or to be excluded). The reason is to support the differential diagnosis by physicians.

The code system and name of diseases in the disease master data are based on our specifications.

The number of records in the disease master data is 1,000.

The total number of diseases is approximately 6,700.

Table 7 shows the Example of the disease master data.

4.1.4. Case data

The case data is used as "training data."

The procedure for making the case data for CDSS is discussed in the following chapter:

2.2. Case data for Clinical Decision Support System

As discussed before, no technology has yet been developed to do these tasks automatically. Therefore, we had to do these tasks manually.

We always weighted the scores for confirmed diseases by α (e.g., α = 10).

The number of data for the case data is around 26,000.

We converted our case data to a data-frame format.

Table 8 shows the Example of the case date (data-frame format).

4.2. Development and execution environment

Table 9 shows the Development and execution environment of the system.

4.3. Neural network configuration

The notation rules for the loss and valuation function are as follows:

- Loss function: UPPER CASE (ex: NDCG, MSE, etc)

- Evaluation function: lower case (ex: ndcg, mse, etc.)

Table 10 shows the Neural network configuration of the system.

4.4. Implementation code

Table 11 shows the Code fragment of the system.

4.5. Figures and Tables

Table 5

Example of expression in the symptom master data
Type of expression	Type of symptom	Example of item	Example of expression
Qualitative expression	Physical findings	Fever Respiratory rate	high / low more / less
	Laboratory test results	Blood glucose level White blood cell count	high / low more / less
Image findings	Imaging test results	Chest radiograph	Abnormal chest radiograph
		Bone radiograph	Abnormal bone radiograph

Table 6

Example of the symptom master data
	Code	Name
1	fever	Fever; high body temperature; hyperthermia
2	head	Headache; nuchal pain; temporal headache; morning headache
3	sore	Sore throat; pharyngeal erythema; pharyngeal erosion; throat pain
4	myalg	Myalgia; muscle (s) ache; muscle pain
5	fatig	Fatigue; easy fatigued/fatigability; tiredness; lassitude; feel tired
6	weigh	Weight loss; the clothes appear to fit loosely
7	arthralg	Arthralgia (take no account of the property or the location)
8	diarrh	Diarrhea; loose stool; pass a loose bowel movement
9	lymphn	Lymphadenopathy/adenopathy (take no account of the property/location)
	…
Citation: symptom master data of our system

Table 7

Example of the disease master data
	Code	Name
1	42	Acute porphyria (acute intermittent porphyria, variegate porphyria, Hereditary coproporphyria), …
2	790	Diabetic coma imminent state, hyperosmolar hyperglycemic syndrome (HHS), …
3	535	Pesticide poisoning, organophosphate toxicity …
4	876	Lead poisoning (almost chronic), aromatic hydrocarbons intoxication, halocarbon poisoning, …
5	117	Cytomegalovirus infection, CMV syndrome (note: AIDS and other opportunistic infections), post-cardiac surgery (extracorporeal circulation) syndrome, …
6	920	Visceral rupture, bleeding or hematoma (cause of anemia, faint, or falling down) infarction, …
7	765	Hyponatremia, …
8	499	Portal vein obstruction, portal vein thrombosis, pylephlebitis, portal vein gas (carefully evaluate the existence of intestinal necrosis), …
9	957	Acetaminophen poisoning, …
	…
Citation: disease master data of our system

Table 8

Example of the case date (data-frame format)
case	X					Y
	0	1		581	582	0	1		998	999
0	1	0		0	0	0.000	1.442		0.385	5.290
1	0	0		0	0	0.000	0.000		10.000	6.250
			…					…
26384	0	0		0	0	1.282	0.961		3.460	1.923
Citation: case data of our system

Table 9

Development and execution environment of the system
Type
Programming Language	Python 3.8.x
Software Libraries	TensorFlow 2.7.x
	TensorFlow Ranking 0.5.x
Web Application Framework	Flask 2.0.x
Cloud Computing Services	Google Cloud Platform
	App Engine
	AI Platform Engine
	Firebase Authentication

Table 10

Neural network configuration of the system
Items	Values		Notes
Input layer	583	units	Number of symptom codes
Hidden layer	1	layer
	1024	units
Output layer	1000	units	Number of disease records
Loss functions	Approximate NDCG loss (A-NDCG)		Our system
	Mean Squared Error (MSE)		Conventional system
	Gumbel Approximate NDCG loss (G-A-NDCG)		For improve prediction performance
Evaluation functions	ndcg
	ndcg@k
	mse

Table 11

Code fragment of the system
import tensorflow as tf import tensorflow_ranking as tfr N_SYMPTOM_CODE = 583 N_DISEASE_CODE = 1000 HIDDEN_UNITS = 1024 ACTIVATION = tf.keras.activations.relu OPTIMIZER = tf.keras.optimizers.Adam() # LOSS = tf.keras.losses.MeanSquaredError() LOSS = tfr.keras.losses.ApproxNDCGLoss() # LOSS = tfr.keras.losses.GumbelApproxNDCGLoss() METRICS = [ tf.keras.metrics.MeanSquaredError(name="mse"), tfr.keras.metrics.NDCGMetric(name="ndcg"), tfr.keras.metrics.NDCGMetric(name="ndcg_1", topn = 1), tfr.keras.metrics.NDCGMetric(name="ndcg_3", topn = 3), tfr.keras.metrics.NDCGMetric(name="ndcg_5", topn = 5), tfr.keras.metrics.NDCGMetric(name="ndcg_10", topn = 10), tfr.keras.metrics.NDCGMetric(name="ndcg_20", topn = 20), ] # BATCH_SIZE = 4096 # MSE BATCH_SIZE = 512 # ApproxNDCG # BATCH_SIZE = 64 # GumbelApproxNDCG # EPOCHS = 900 # MSE EPOCHS = 80 # ApproxNDCG # EPOCHS = 80 # GumbelApproxNDCG def train(X_train, y_train, X_test, y_test): model = tf.keras.models.Sequential() model.add(tf.keras.layers.InputLayer(input_shape=(N_SYMPTOM_CODE,))) model.add(tf.keras.layers.Dense(HIDDEN_UNITS)) model.add(tf.keras.layers.Activation(ACTIVATION)) model.add(tf.keras.layers.Dense(N_DISEASE_CODE)) model.compile(OPTIMIZER, LOSS, METRICS) model.fit(X_train, y_train, batch_size = BATCH_SIZE, epochs = EPOCHS, validation_data=(X_test, y_test)) return model

5.1. Evaluation purposes

Evaluation purposes for the Machine Learning (ML) performance and the Differential Diagnostic (DDx) performance are to demonstrate the following performance:

- The ML performance of the system is superior to the conventional system.

- The DDx performance of the system is superior to the conventional system.

- The DDx performance of the system is useful to support the DDx process by physicians.

- The CDSS is useful for preventing diagnostic errors by physicians.

5.2. Evaluated Performances

- Machine Learning (ML) performances

- Learning curves

- Values of the evaluation function

- Differential Diagnosis (DDx) performances with actual cases

- Inputted symptoms and predicted diseases

- Inputted symptoms and the target disease's ranking

5.3. Compared system

Comparison for the system was the conventional system. The conventional systems are one generation before our system. [15]

Similarly, the conventional system uses Learning-to-Rank (LTR). However, LTR for the conventional system is the pointwise approach. The loss function of the conventional system is Mean Squared Error (MSE).

5.4. Conditions for evaluation and comparison

Except for the loss functions, the case data, hyperparameters, neural network configurations, and other conditions were the same.

5.5. Evaluation criteria for Differential Diagnostic performances

As evaluation criteria for DDx performances, we focused on confirmed diseases (or related diseases) that should be ranked in the top 10th predicted diseases.

The reasons are:

- The DDx process by physicians is a Bayesian game.

- A Bayesian game is a kind of incomplete information game.

- In the DDx process by physicians, the acquired information, thoughts, and knowledge may contain mistakes or omissions. [20]

- The main objective of today's CDSS is to be a Decision Support System, not a Diagnosis System.

- Using the predicted diseases of CDSS as a reference, physicians decide the final confirmed disease(s) by themselves.

6.1. Evaluation method

The Machine Learning (ML) performance evaluated was as follows:

- Learning curves

- Value of evaluation function

We evaluated the ML performance on k-fold cross-validation (k = 5).

6.2. Evaluation results and discussion

6.2.1. Loss functions: A-NDCG vs. MSE

Due to memory space requirements when using A-NDCG, the batch size for training is set to 512.

Figure 2 shows the Learning curves of ndcg: A-NDCG vs. MSE.

Figure 3 shows the Learning curves of mse: A-NDCG vs. MSE.

Table 12 shows the Value of evaluation functions: A-NDCG vs. MSE.

The number of epochs in training was larger for MSE.

The training time was longer for A-NDCG.

The memory space requirement was larger for A-NDCG.

The values of the evaluation functions (ndcg and ndcg@k) were always higher for A-NDCG.

The ML performance differences between A-NDCG and MSE were very significant.

We found that the prediction model with A-NDCG tended to overfit.

Note that we do not think the evaluation function mse (Mean Squared Error) is appropriate for LTR with A-NDCG.

6.2.2. Loss functions: G- A-NDCG vs. A-NDCG64

Due to the memory space requirements when using G-A-NDCG, the batch size for training is set to 64.

From now, Approximate NDCG loss (Batch size = 64) is referred to as "A-NDCG64."

Figure 4 shows the Learning curves of ndcg: G-A-NDCG vs. A-NDCG64.

Figure 5 shows the Learning curves of mse: G-A-NDCG vs. A-NDCG64

Table 13 shows the Value of evaluation functions: G-A-NDCG vs. A-NDCG64.

The number of epochs in training for both was approximately equal.

The training time was much longer for G-A-NDCG.

The memory space requirement was much larger for G-A-NDCG.

The values of the evaluation functions (ndcg and ndcg@k) were tending higher for G-A-NDCG.

The ML performance differences between G-A-NDCG and A-NDCG were not significant.

We assume this is due to the memory space limits of GPU for our environment.

As with A-NDCG, we found that the prediction model with G-A-NDCG tended to overfit.

Like the previous result, we do not think the evaluation function mse (Mean Squared Error) is more appropriate for LTR with G-A-NDCG.

6.3. Figures and Tables

Figure 2 Learning curves of ndcg: A-NDCG vs. MSE

Evaluation function: ndcg

Loss functions: Approximate NDCG loss (A-NDCG)

Mean Squared Error (MSE)

Batch size: 512

Figure 3 Learning curves of mse: A-NDCG vs. MSE

Evaluation function: mse

Loss functions: Approximate NDCG loss (A-NDCG)

Mean Squared Error (MSE)

Batch size: 512

Figure 4 Learning curves of ndcg: G-A-NDCG vs. A-NDCG64

Evaluation function: ndcg

Loss functions: Gumbel Approximate NDCG loss(G-A-NDCG)

Approximate NDCG loss(A-NDCG64)

Batch size: 64

Figure 5 Learning curves of mse: G-A-NDCG vs. A-NDCG64

Evaluation function: mse

Loss functions: Gumbel Approximate NDCG loss(G-A-NDCG)

Approximate NDCG loss (A-NDCG64)

Batch size: 64

Table 12

Value of evaluation functions: A-NDCG vs. MSE
Loss functions	Batch size	Evaluation functions
		ndcg	ndcg@5	ndcg@10	ndcg@20
A-NDCG	512	0.7098	0.6205	0.6485	0.6680
MSE	512	0.5835	0.4470	0.4845	0.5139

Table 13

Value of evaluation functions: G-A-NDCG vs. A-NDCG64
Loss functions	Batch size	Evaluation functions
		ndcg	ndcg@5	ndcg@10	ndcg@20
G-A-NDCG	64	0.7014	0.6113	0.6394	0.6585
A-NDCG64	64	0.7016	0.6106	0.6375	0.6375

7.1. Evaluation methods

The Differential Diagnosis (DDx) performance evaluated was as follows:

- Inputted symptoms and predicted diseases

- Inputted symptoms and the target disease's ranking

In this study, the evaluation method for DDx performance differs from previous articles (compared subjects: Physicians vs. CDSS; target cases: Actual cases in clinical practice).

The reasons are:

- The compared subjects: Our CDSS vs. the conventional CDSS

- The target cases: Rare and difficult-to-diagnose cases in clinical practice

For this reason, we used the cases from case reports adapted for our purposes rather than actual cases.

The case citation was from case reports of "The New England Journal of Medicine (NEJM)."

The cases used in the evaluation were as follows:

- Disease with characteristic symptoms

- Acute intermittent porphyria [21]

- Difficult-to-diagnose disease with few characteristic symptoms

- Acute HIV-1 infection [22]

- Case with diagnostic errors

- Subacute bacterial endocarditis caused by bartonella [23]

- DDx performance using the Gumbel Approximate NDCG loss

- Acute intermittent porphyria [21]

- Acute HIV-1 infection [22]

In the interest of fairness, these cases were not used when training the model.

7.2. Evaluation results and Discussion

7.2.1. Disease with characteristic symptoms

We evaluated the Differential Diagnostic (DDx) performance of the disease with characteristic symptoms.

The DDx of these diseases is manageable to a conventional CDSS.

The case we used was acute intermittent porphyria (AIP). [21]

In this system, the confirmed disease, in this case, is as follows:

- Acute intermittent porphyria (AIP)

Table 14 and Table 15 show the Inputted symptoms and predicted diseases.

Table 16 shows the Inputted symptoms and the target disease's ranking.

In both systems, the predicted ranking of confirmed disease was 1st

In the predicted diseases of the system (A-NDCG), the excluded diseases for AIP (ex: lead poisoning) were listed at the top of the list. [24], [25]

In this case, the predicted diseases of the system provided useful information for the DDx process by physicians.

Regarding "Inputted symptoms and the target disease's ranking," in both systems, at the point where the characteristic symptoms (hyponatremia and abnormal liver function) were inputted, the final confirmed disease was listed at the top of the list.

For the DDx of diseases with characteristic symptoms, we believe that the DDx performances of both systems are not significantly different.

7.2.2. Difficult-to-diagnose disease with few characteristic symptoms

We evaluated the Differential Diagnosis (DDx) performance of the difficult-to-diagnose disease with few characteristic symptoms.

The DDx of these diseases is difficult to conventional CDSS.

The case we used was acute HIV-1 infection. [22]

In HIV infection, acute meningitis symptoms may develop at the time of initial infection. [26]

In this system, the related diseases, including the confirmed disease, in this case, are as follows:

- Acute HIV-1 infection

- Acute viral meningitis

Therefore, these diseases were also defined as related diseases to confirmed diseases.

Table 17 and Table 18 show the Inputted symptoms and predicted diseases.

Table 19 shows the Inputted symptoms and the target disease's ranking.

In this system (A-NDCG), the predicted rankings of related diseases were as follows:

1st Acute HIV-1 infection

3rd Acute viral meningitis

However, in the conventional system (MSE), the predicted rankings of related diseases were less than the 20th.

Regarding "Inputted symptoms and the target disease's ranking," in this system (A-NDCG), at the point where few symptoms were inputted, related diseases were listed at the top of the list.

In this case, many of these symptoms are common in other diseases.

For the DDx of difficult-to-diagnose diseases with few characteristic symptoms, we believe that the DDx performances of the system are superior.

7.2.3. Case with diagnostic errors

Cognitive biases, such as confirmation bias, are among the most frequent causes of diagnostic errors. [27]

We believe that CDSS is useful for preventing diagnostic errors.

We evaluated the Differential Diagnostic (DDx) performance of a case with diagnostic errors.

The final confirmed disease of the case was subacute bacterial endocarditis caused by bartonella. [23]

The title of the case report is "Copycat." In this case, this patient had a history of HCV infection. Initially, due to confirmation bias, the case report's authors did not focus on the characteristic symptoms of endocarditis (heart murmur, purpura, etc.) but this HCV infection. As a result, they reported the misdiagnosed case as mixed cryoglobulinemia by HCV.

In this system, the related diseases, including the confirmed disease, in this case, are as follows:

- Subacute bacterial endocarditis (SBE)

- Acute bacterial endocarditis

- Infectious endocarditis

Therefore, these diseases were also defined as related diseases to confirmed diseases.

In addition, the misdiagnosed disease is as follows:

- Mixed cryoglobulinemia

Table 20, Table 21, Table 22, and Table 23 show the Inputted symptoms and predicted diseases.

We believe the causes of these results are biased information inputs by confirmation bias.

Despite the biased information, the system always listed the related disease at the top of the list. In the DDx process by physicians, if they had this information, we assume that their differential disease list would include not only HIV infection but also SBE.

We believe that the CDSS, including our system, will prevent diagnostic errors by physicians.

7.2.4. Differential diagnostic performance using the Gumbel Approximate NDCG loss

To evaluate the Differential Diagnostic (DDx) performance using Gumbel Approximate NDCG loss (G-A-NDCG), we compared the performance of G-A-NDCG and Approximate NDCG loss (A-NDCG).

The cases we evaluated were the following cases used before:

- Acute intermittent porphyria (AIP) [21]

- Acute HIV-1 infection [22]

Due to the memory space requirements when using G-A-NDCG, the batch size for training is set to 64.

Approximate NDCG loss (Batch size = 64) is referred to as "A-NDCG64."

Table 24, Table 25, Table 26, and Table 27 show the Inputted symptoms and predicted diseases.

From the predicted diseases of these cases, at this time, we do not believe there is a difference in DDx performance between G-A-NDCG and A-NDCG64.

As discussed before, due to the memory space requirements, we assume that the improvement of DDx performance when using G-A-NDCG was limited.

7.3. Figures and Tables

Table 14 Inputted symptoms and predicted diseases:

Acute intermittent porphyria:

A-NDCG

	Inputted symptoms			Scores	Predicted diseases
1	Abdominal pain		1	1.41	Acute intermittent porphyria
2	Constipation		2	1.21	Diabetic coma imminent state
3	Hyponatremia		3	0.56	Pesticide poisoning, Organophosphate toxicity
4	Abdominal tenderness		4	0.51	Lead poisoning (almost chronic)
5	Elevated serum liver enzyme		5	0.45	Heat stroke (hyperthermia)
6	Tachycardia	→	6	0.31	Cytomegalovirus infection
7	Photophobia		7	0.25	Visceral rupture
8	Thirsty		8	0.16	Hyponatremia
9	Borborygmus		9	0.13	Portal vein obstruction
10	Dehydration		10	0.12	Acetaminophen poisoning
11	Confusion				...
Case: acute intermittent porphyria [21]

Loss function: Approximate NDCG loss (A-NDCG)

Table 15 Inputted symptoms and predicted diseases:

Acute intermittent porphyria:

MSE

	Inputted symptoms			Scores	Predicted diseases
1	Abdominal pain		1	9.25	Acute intermittent porphyria
2	Constipation		2	8.17	Enterohemorrhagic e. coli (EHEC) infection
3	Hyponatremia		3	7.64	Visceral rupture
4	Abdominal tenderness		4	7.56	Fibromyalgia (fibrositis)
5	Elevated serum liver enzyme		5	7.53	Cancerous peritonitis
6	Tachycardia	→	6	7.29	Withdrawal symptoms of alcohol and drugs
7	Photophobia		7	7.28	Colorectal cancer
8	Thirsty		8	7.26	Irritable bowel syndrome, Functional dyspepsia (FD)
9	Borborygmus		9	7.23	Drugs (laxatives, etc.)
10	Dehydration		10	7.07	Eating disorder
11	Confusion				...
Case: acute intermittent porphyria [21]

Loss function: Mean Squared Error (MSE)

Table 16

Inputted symptoms and the target disease's ranking: Acute intermittent porphyria
	Inputted symptoms	Ranking
		Target disease	AIP
		Loss function	A-NDCG	MSE
1	Abdominal pain		---	---
2	Constipation	→	---	---
3	Hyponatremia		1	4
4	Abdominal tenderness	→→	1	19
5	Elevated serum liver enzyme		1	4
6	Tachycardia	→→→	1	1
7	Photophobia		1	1
8	Thirsty	→→→→	1	1
9	Borborygmus		1	1
10	Dehydration	→→→→→	1	1
11	Confusion		1	1
Case: acute intermittent porphyria [21]

Target disease: acute intermittent porphyria (AIP)

Loss functions: Approximate NDCG loss (A-NDCG)

Mean Squared Error (MSE)

---: 20th and lower

Table 17 Inputted symptoms and predicted diseases:

Acute HIV-1 infection:

A-NDCG

	Inputted symptoms			Scores	Predicted diseases
1	Fever		1	1.61	Acute HIV-1 infection
2	Headache		2	1.51	Polyneuropathy
3	Sore throat		3	0.91	Acute viral meningitis
4	Consciousness indistinctness		4	0.88	West Nile fever
5	Chills		5	0.77	Cat-scratch disease
6	Muscles ache		6	0.46	Acute Q fever
7	Swallowing pain	→	7	0.23	Epidemic hepatitis A
8	Pharyngolaryngeal abnormality		8	0.21	Chronic fatigue syndrome
9	Aphasia		9	0.13	Sepsis
10	Apraxia		10	0.12	Toxoplasmosis
11	Fatigue				...
12	Muscle weakness
13	Anorexia
14	Weight loss
15	Dementia
Case: acute HIV-1 infection [22]

Loss function: Approximate NDCG loss (A-NDCG)

Table 18 Inputted symptoms and predicted diseases:

Acute HIV-1 infection:

MSE

	Inputted symptoms			Scores	Predicted diseases
1	Fever		1	10.16	Epidemic hepatitis A
2	Headache		2	9.93	Acute Q fever
3	Sore throat		3	9.75	Acute pharyngitis
4	Consciousness indistinctness		4	9.66	Polyneuropathy
5	Chills		5	9.28	Lymphocytic choriomeningitis
6	Muscles ache		6	9.17	Herpes labialis
7	Swallowing pain	→	7	8.99	Side effects of interferon
8	Pharyngolaryngeal abnormality		8	8.86	Sepsis
9	Aphasia		9	8.74	Chronic fatigue syndrome
10	Apraxia		10	8.73	Retropharyngeal infection
11	Fatigue				...
12	Muscle weakness
13	Anorexia
14	Weight loss
15	Dementia
Case: acute HIV-1 infection [22]

Loss function: Mean Squared Error (MSE)

Table 19

Inputted symptoms and the target disease's ranking: Acute HIV-1 infection
	Inputted symptoms	Ranking
		Target diseases	HIV-1		AVM
		Loss functions	A-NDCG	MSE	A-NDCG	A-NDCG
1	Fever		---	---	---	---
2	Headache		---	---	9	1
3	Sore throat		14	---	10	---
4	Consciousness indistinctness	→	---	---	3	10
5	Chills		---	---	2	9
6	Muscles ache	→→	---	---	3	---
7	Swallowing pain		14	---	5	---
8	Pharyngolaryngeal abnormality	→→→	---	---	9	---
9	Aphasia		---	---	6	---
10	Apraxia	→→→→	---	---	2	19
11	Fatigue		---	---	2	---
12	Muscle weakness	→→→→→	12	---	2	---
13	Anorexia		12	---	1	---
14	Weight loss		1	---	3	---
15	Dementia		1	---	3	---
Case: acute HIV-1 infection [22]

Target diseases: acute HIV-1 infection (HIV-1)

acute viral meningitis (AVM)

Loss functions: Approximate NDCG loss (A-NDCG)

Mean Squared Error (MSE)

---: 20th and lower

Table 20 Inputted symptoms and predicted diseases:

Subacute bacterial endocarditis caused by bartonella:

In progress: n = 5

	Inputted symptoms			Scores	Predicted diseases
1	Weight loss		1	0.78	Hodgkin lymphoma
2	Itching		2	0.60	Heart tumor
3	Fatigue		3	0.41	Mitral valve stenosis
4	Tachypnea		4	0.32	Infectious endocarditis
5	Heart murmur		5	0.17	Aortic stenosis
6			6	0.16	Acute Q fever
7			7	0.04	Hyperthyroidism
8			8	0.03	Immunoblastic lymphadenopathy
9		→	9	0.03	Hepatic amyloidosis
10			10	0.03	Hashimoto's disease and autoimmune thyroiditis
11					…
12
13
14
15
16
17
18
Case: subacute bacterial endocarditis caused by bartonella [23]

Loss functions: Approximate NDCG loss (A-NDCG)

In progress: n = 5

Table 21 Inputted symptoms and predicted diseases:

Subacute bacterial endocarditis caused by bartonella:

In progress: n = 9

	Inputted symptoms			Scores	Predicted diseases
1	Weight loss		1	1.61	Zieve syndrome
2	Itching		2	1.35	Disseminated intravascular coagulation
3	Fatigue		3	0.94	Chronic hepatitis
4	Tachypnea		4	0.77	Wilson's disease
5	Heart murmur		5	0.76	Acute hepatitis
6	Purpura		6	0.69	Hepatic amyloidosis
7	Anemia		7	0.59	Infectious endocarditis
8	Elevated serum liver enzyme		8	0.59	(Compensated/ uncompensated) liver cirrhosis
9	Hyperbilirubinemia	→	9	0.53	Subacute bacterial endocarditis
10			10	0.53	Gastric cancer
11					…
12
13
14
15
16
17
18
Case: subacute bacterial endocarditis caused by bartonella [23]

Loss functions: Approximate NDCG loss (A-NDCG)

In progress: n = 9

Table 22 Inputted symptoms and predicted diseases:

Subacute bacterial endocarditis caused by bartonella:

In progress: n = 14

	Inputted symptoms			Scores	Predicted diseases
1	Weight loss		1	1.78	Subacute bacterial endocarditis
2	Itching		2	1.35	Hepatic amyloidosis
3	Fatigue		3	1.32	Chronic hepatitis
4	Tachypnea		4	1.29	Infectious endocarditis
5	Heart murmur		5	1.16	Disseminated intravascular coagulation
6	Purpura		6	1.15	Mixed cryoglobulinemia
7	Anemia		7	1.10	Acute bacterial endocarditis
8	Elevated serum liver enzyme		8	0.99	Idiopathic (or secondary) retroperitoneal fibrosis
9	Hyperbilirubinemia	→	9	0.98	Tuberculosis or syphilis
10	ESR elevation		10	0.90	Thrombotic thrombocytopenic purpura
11	RA/RF positive				…
12	Hematuria
13	Bacteriuria
14	Proteinuria
15
16
17
18
Case: subacute bacterial endocarditis caused by bartonella [23]

Loss functions: Approximate NDCG loss (A-NDCG)

In progress: n = 14

Table 23 Inputted symptoms and predicted diseases:

Subacute bacterial endocarditis caused by bartonella:

Final: n = 18

	Inputted symptoms			Scores	Predicted diseases
1	Weight loss		1	2.70	Mixed cryoglobulinemia
2	Itching		2	2.62	Chronic hepatitis
3	Fatigue		3	2.03	Subacute bacterial endocarditis
4	Tachypnea		4	1.48	Hepatic amyloidosis
5	Heart murmur		5	1.47	Rapidly progressive glomerulonephritis syndrome
6	Purpura		6	1.34	Acute bacterial endocarditis
7	Anemia		7	1.27	Infectious endocarditis
8	Elevated serum liver enzyme		8	0.94	Polyarteritis nodosa
9	Hyperbilirubinemia	→	9	0.78	Autoimmune hemolytic anemia
10	ESR elevation		10	0.75	Disseminated intravascular coagulation
11	RA/RF positive				…
12	Hematuria
13	Bacteriuria
14	Proteinuria
15	Hypocomplementemia
16	Angiitis
17	Nephritis
18	Animal Origin Infections
Case: subacute bacterial endocarditis caused by bartonella [23]

Loss functions: Approximate NDCG loss (A-NDCG)

Final: n = 18

Table 24 Inputted symptoms and predicted diseases:

Acute intermittent porphyria:

G-A-NDCG

	Inputted symptoms			Scores	Predicted diseases
1	Abdominal pain		1	5.86	Acute intermittent porphyria
2	Constipation		2	5.46	Diabetic coma imminent state
3	Hyponatremia		3	2.73	Heat stroke (hyperthermia)
4	Abdominal tenderness		4	-0.64	Lead poisoning (almost chronic)
5	Elevated serum liver enzyme		5	-2.97	Drug side effects
6	Tachycardia	→	6	-3.85	Hyponatremia
7	Photophobia		7	-4.01	Intussusception
8	Thirsty		8	-4.51	Genuine epilepsy
9	Borborygmus		9	-4.57	Visceral rupture
10	Dehydration		10	-4.75	Dumping syndrome
11	Confusion				…
Case: acute intermittent porphyria [21]

Loss functions: Gumbel Approximate NDCG loss (A-NDCG)

Batch size: 64

Table 25 Inputted symptoms and predicted diseases:

Acute intermittent porphyria:

A-NDCG64

	Inputted symptoms			Scores	Predicted diseases
1	Abdominal pain		1	4.88	Acute intermittent porphyria
2	Constipation		2	0.31	Diabetic coma imminent state
3	Hyponatremia		3	-0.15	Visceral rupture
4	Abdominal tenderness		4	-1.92	Heat stroke (hyperthermia)
5	Elevated serum liver enzyme		5	-2.19	(Tumor-related) syndrome of inappropriate secretion of ADH
6	Tachycardia	→	6	-2.45	Pesticide poisoning, organophosphate toxicity
7	Photophobia		7	-2.59	Lead poisoning (almost chronic)
8	Thirsty		8	-2.90	Acetaminophen poisoning
9	Borborygmus		9	-4.01	Hyponatremia
10	Dehydration		10	-4.59	Withdrawal symptoms of alcohol and drugs
11	Confusion				…
Case: acute intermittent porphyria [21]

Loss functions: Approximate NDCG loss (A-NDCG64)

Batch size: 64

Table 26 Inputted symptoms and predicted diseases:

Acute HIV-1 infection:

G-A-NDCG

	Inputted symptoms			Scores	Predicted diseases
1	Fever		1	0.74	Acute HIV-1 infection
2	Headache		2	-0.95	Herpes labialis
3	Sore throat		3	-0.96	(Idiopathic) normal pressure hydrocephalus
4	Consciousness indistinctness		4	-1.18	Histoplasmosis
5	Chills		5	-3.34	Polyneuropathy
6	Muscles ache		6	-3.86	Polymyalgia rheumatica
7	Swallowing pain	→	7	-4.85	West Nile fever
8	Pharyngolaryngeal abnormality		8	-5.16	Side effects of interferon
9	Aphasia		9	-5.51	Chronic fatigue syndrome
10	Apraxia		10	-5.58	Cat-scratch disease
11	Fatigue				…
12	Muscle weakness
13	Anorexia
14	Weight loss
15	Dementia
Case: acute HIV-1 infection [22]

Loss functions: Gumbel Approximate NDCG loss (A-NDCG)

Batch size: 64

Table 27 Inputted symptoms and predicted diseases:

Acute HIV-1 infection:

A-NDCG64

	Inputted symptoms			Scores	Predicted diseases
1	Fever		1	1.39	Acute HIV-1 infection
2	Headache		2	-0.25	West Nile fever
3	Sore throat		3	-0.36	Sepsis
4	Consciousness indistinctness		4	-1.45	Polyneuropathy
5	Chills		5	-2.74	Chronic subdural hematoma
6	Muscles ache		6	-2.94	Meningitis or encephalitis
7	Swallowing pain	→	7	-3.08	Cat-scratch disease
8	Pharyngolaryngeal abnormality		8	-3.15	Lymphocytic choriomeningitis
9	Aphasia		9	-4.28	Autonomic imbalance
10	Apraxia		10	-4.53	Addison's disease
11	Fatigue				…
12	Muscle weakness
13	Anorexia
14	Weight loss
15	Dementia
Case: acute HIV-1 infection [22]

Loss functions: Approximate NDCG loss (A-NDCG)

Batch size: 64

We are researching, developing, and publishing the Diagnostic Decision Support System (CDSS) based on Learning-to-Rank (LTR).

This paper discusses our system's design, implementation, and evaluation.

The main objectives of the system are as follows:

- Supporting differential diagnoses by internists and general practitioners

- Preventing diagnostic errors by physicians

The main features of the system are as follows:

A physician inputs a patient’s symptoms, findings, and test results to the system,

and the system outputs a ranking list of possible diseases.

8.1. Evaluation results

We evaluated Machine Learning (ML) performance and Differential Diagnosis (DDx) performance.

The ML and DDx performance of the system (listwise approach: A-NDCG) was higher than that of the conventional system (pointwise approach: MSE).

As loss functions, we evaluated the ML and DDx performance of Gumbel Approximate NDCG loss (G-A-NDCG) and Approximate NDCG loss (G-NDCG). The ML performance of G-A-NDCG was higher than that of A-NDCG. However, the DDx performance was not different. Due to the memory space requirements, we assume that the improvement of prediction performance when using G-A-NDCG is limited.

In terms of both ML and DDx performance, we have demonstrated that the CDSS is useful for physicians to support DDx and prevent diagnostic errors.

8.2. Differential Diagnosis process by physicians and Learning to Rank by machines

The prediction algorithm of the system is Learning-to-Rank (LTR). The DDx process by physicians is an iterative process with Recalling, Refining, and Ranking differential diseases.

8.3. Case data and Information Retrieval

Our system's case data (= training data) and predicted results are almost the same data structure.

Table 28 shows the Case data and predicted results of our system.

As a result, if experienced physicians validate the predicted results, compared to the conventional system with a pointwise approach, we believe that our system with a listwise approach is easier to provide feedback to the prediction model on its validation results.

As discussed before, no technology has yet been developed to optimize case data for a listwise approach automatically.

Therefore, we had to do these tasks manually (and by only one physician).

As a result, due to his knowledge and thought, we think that our system may have both bias and outstanding performance.

For the practical application of CDSS, we believe that developing the following Information Technologies (IT) is necessary:

- Technology for predicting diseases, such as Learning-to-Rank (LTR)

- Technology for text-mining information on diseases from literature

- Technology for converting text-mining data to metadata

For this purpose, we believe using Information Retrieval (IR) technologies is effective.

8.4. Potentials for Clinical Decision Support System

According to our experience and knowledge, we believe that Clinical Decision Support System (CDSS), including our system, has the following potential:

- Recall rare diseases

- Support differential diagnoses for difficult-to-diagnose diseases

- Prevent diagnostic errors

8.5. Evolution into Explainable Clinical Decision Support System

We believe that our system has the potential for evolution to an Explainable Clinical Decision Support System (X-CDSS). [28]

The reasons for this are as follows:

- The affinity between DDx processes by experienced physicians and LTR (listwise approach)

- The similarity between case data (= training data) and predicted results

- The simple neural network

- The number of internal hiding layers is one.

- The number of learnable times (epochs) is relatively small.

We will continue to develop the Ultimate Clinical Decision Support System (U-CDSS).

8.6. Figures and Tables

Table 28

Case data and predicted results of our system
	X: Explanatory variables	y: Explained variables
Case data (= Training data)	Observed symptoms	Confirmed disease(s) and those score(s) Differential diseases (related or to be excluded) and those scores
Predicted results	Inputted symptoms	Predicted diseases and those scores

CDSS: Clinical Decision Support System
DDSS: Diagnosis Decision Support System
IR: Information Retrieval
LTR: Learning to rank
ML: Machine Learning
DDx: Differential Diagnosis
NDCG: Normalized Discounted Cumulative Gain
A-NDCG: Approximate NDCG, as a loss function
A-NDCG64: Approximate NDCG, as a loss function (Batch size = 64)
G-A-NDCG: Gumbel Approximate NDCG, as a loss function
MSE: Mean Squared Error, as a loss function
ndcg: NDCG, as an evaluation function
mse: Mean Squared Error, as an evaluation function

10.1 Ethics approval and consent to participate

Not applicable

10.2 Consent for publication

Not applicable

10.3 Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Contact information is as follow:

https://www.diagnosis.or.jp/

mailto:[email protected]

License information is as follows:

CC BY-NC-SA 4.0

10.4 Competing interests

All authors declare that they have no competing interests.

10.5 Funding

Not applicable

10.6 Authors' contributions

Miyachi, Yasuhiko: Corresponding author,
Improved system's developer

Ishii, Osamu: Original system's developer

Dr. Torigoe, Keijiro: Chief authority,
Medical supervisor

All authors read the final manuscript, and approved it for submission.

10.7 Acknowledgements

Not applicable

Miyachi Y, Torigoe K, Ishii O. Computer-Aided Decision Support System based on LTR algorithm - Collaboration of a clinician and the machine learning in the differential diagnosis -. The 41st Joint Conference on Medical Informatics (The 22th Annual Meeting of JAMI) [Internet]. 2021 Nov 18 [cited 2022 Jul 24]; 41st:801–6. Available from: https://jglobal.jst.go.jp/detail?JGLOBAL_ID=202102273407233811
Miyachi Y, Torigoe K, Ishii O. Clinical Decision Support System based on Learning to Rank - Improving diagnostic performance with Pointwise approach to Listwise approach -. In: The 36th Annual Conference of the Japanese Society for Artificial Intelligence, 2022 [Internet]. 2022 [cited 2022 Jul 24]. Available from: https://www.jstage.jst.go.jp/article/pjsai/JSAI2022/0/JSAI2022_4M1GS1001/_article/-char/ja/
Kohn LT, Corrigan JM, Molla S. To Err Is Human. 1999; 6. Available from: https://www.ncbi.nlm.nih.gov/books/NBK225182/
Balogh EP, Miller BT, Ball JR. Improving Diagnosis in Health Care. 2016. Available from: https://www.ncbi.nlm.nih.gov/books/NBK338596/
Shimizu T. Perspective: Al in diagnostic medicine. Japanese Journal of Allergology.
2020; 69(8). Available from: https://www.jstage.jst.go.jp/article/arerugi/69/8/69_658/_article/-char/ja/4
Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. Vol. 3, npj Digital Medicine. 2020. Available from: https://www.nature.com/articles/s41746-020-0221-y
Stern S, Cifu A, Altkorn D. Symptom to Diagnosis: An Evidence-Based Guide, 4th Edition. 2020 Available from: https://accessmedicine.mhmedical.com/book.aspx?bookID=2715
Liu TY. Learning to rank for Information Retrieval. 2009. Available from: https://dl.acm.org/doi/10.1561/1500000016
Berner ES. Clinical Decision Support Systems: Theory and Practice, Third Edition.
2016. Available from: https://link.springer.com/book/10.1007/978-3-319-31913-1
Schwartz A, Elstein AS. Clinical Problem Solving and Diagnostic Decision Making: A Selective Review of the Cognitive Research Literature. In: The Evidence Base of Clinical Diagnosis: Theory and methods of diagnostic research: Second Edition. 2009. Available from: https://pubmed.ncbi.nlm.nih.gov/11909793/
Differential Diagnosis Tool | Isabel Healthcar [Internet]. [cited 2022 Aug 7]. Available from: https://www.isabelhealthcare.com/
The Laboratory of Computer Science | DXplain [Internet]. [cited 2022 Aug 7]. Available from: http://www.mghlcs.org/projects/dxplain/
VisualDx | Visual Clinical Decision Support System (CDSS) [Internet]. [cited 2022 Aug 7]. Available from: https://www.visualdx.com/
J-CaseMap [Internet]. [cited 2022 Aug 8]. Available from: https://www.naika.or.jp/j-casemap/
Kuriyamaa Y, Sota Y, Yano A, Hideki Y, Ishii O, Saio T, et al. Better diagnostic performance using computer-assisted diagnostic support systems in internal medicine. Journal of Okayama Medical Association) [Internet]. 2019. Available from: https://www.jstage.jst.go.jp/article/joma/131/1/131_29/_article/-char/ja/
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. 2019. Available from: https://arxiv.org/abs/1603.04467
Pasumarthi RK, Bruch S, Wang X, Li C, Bendersky M, Najork M, et al. TF-ranking: Scalable tensorflow library for learning-to-rank. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining [Internet]. 2019 [cited 2022 Aug 15]. Available from: https://arxiv.org/pdf/1812.00073
Bruch S, Zoghi M, Bendersky M, Najork M. Revisiting approximate metric optimization in the age of deep neural networks. In: SIGIR 2019 - Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019. Available from: https://dl.acm.org/doi/10.1145/3331184.3331347
Bruch S, Han S, Bendersky M, Najork M. A stochastic treatment of learning to rank scoring functions. In: WSDM 2020 - Proceedings of the 13th International Conference on Web Search and Data Mining. 2020. Available from: https://dl.acm.org/doi/10.1145/3336191.3371844
Richens JG, Lee CM, Johri S. Improving the accuracy of medical diagnosis with causal machine learning. Nat Commun. 2020; 11(1). Available from: https://www.nature.com/articles/s41467-020-17419-7
Fredrick TW, Neto MBB, Johnsrud DO, Camilleri M, Chedid VG. Turning Purple with Pain. New England Journal of Medicine. 2021; 385(6). Available from: https://www.nejm.org/doi/full/10.1056/NEJMcps2105278
Goldstein RH, Mehan WA, Hutchison B, Robbins GK. Case 24-2021: A 63-Year-Old Woman with Fever, Sore Throat, and Confusion. New England Journal of Medicine. 2021; 385(7). Available from: https://www.nejm.org/doi/full/10.1056/NEJMcpc2107345
Dietz BW, Winston LG, Koehler JE, Margaretten M. Copycat. New England Journal of Medicine. 2021; 385(19). Available from: http://www.nejm.org/doi/10.1056/NEJMcps2108885
Tsai MT, Huang SY, Cheng SY. Lead Poisoning Can Be Easily Misdiagnosed as Acute Porphyria and Nonspecific Abdominal Pain. Case Rep Emerg Med. 2017. Available from: https://pubmed.ncbi.nlm.nih.gov/28630774/
Indika NLR, Kesavan T, Dilanthi HW, Jayasena KLSPKM, Chandrasiri NDPD, Jayasinghe IN, et al. Many pitfalls in diagnosis of acute intermittent porphyria: A case report. BMC Res Notes. 2018; 11(1). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6071335/
Park BJ, Wannemuehler KA, Marston BJ, Govender N, Pappas PG, Chiller TM. Estimation of the current global burden of cryptococcal meningitis among persons living with HIV/AIDS. AIDS. 2009; 23(4). Available from: https://pubmed.ncbi.nlm.nih.gov/19182676/
Saposnik G, Redelmeier D, Ruff CC, Tobler PN. Cognitive biases associated with medical decisions: a systematic review. BMC Medical Informatics and Decision Making. 2016;16(1). Available from:
https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-016-0377-1
Schoonderwoerd TAJ, Jorritsma W, Neerincx MA, van den Bosch K.
Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. International Journal of Human Computer Studies. 2021; 154. Available from: https://dl.acm.org/doi/10.1016/j.ijhcs.2021.102684

No competing interests reported.

Design, Implementation, and Evaluation of the Computer-aided Clinical Decision Support System based on Learning-to-Rank: Collaboration between physicians and machine learning in the differential diagnosis process

Status:

Journal Publication

Version 2

Abstract

Background

Method

Results

Conclusions

Figures

1. Introduction

1.1 Diagnostic errors and Clinical Decision Support System

1.2 Main objectives of the Clinical Decision Support System

1.3 Main features of the Clinical Decision Support System

1.4 Example of the Clinical Decision Support System

1.5 Figures and Tables

2. Background

2.1. Differential Diagnosis process by physicians and Learning-to-Rank by machines

2.2. Case data for Clinical Decision Support System

2.3. Information Retrieval and Clinical Decision Support System

2.4. Conventional Clinical Decision Support Systems

2.5. Figures and Tables

3. Design

3.1. Design principles

3.2. Library for Learning-to-Rank

3.3. Case date for Learning-to-Rank with the listwise approach

3.4. Figures and Tables

4. Implementation

4.1. Database

4.1.1. Overview

4.1.2. Symptom master data

4.1.3. Disease master data

4.1.4. Case data

2.2. Case data for Clinical Decision Support System

4.2. Development and execution environment

4.3. Neural network configuration

4.4. Implementation code

4.5. Figures and Tables

5. Evaluation

5.1. Evaluation purposes

5.3. Compared system

5.4. Conditions for evaluation and comparison

5.5. Evaluation criteria for Differential Diagnostic performances

6. Evaluation: Machine Learning Performances

6.1. Evaluation method

6.2. Evaluation results and discussion

6.2.1. Loss functions: A-NDCG vs. MSE

6.2.2. Loss functions: G- A-NDCG vs. A-NDCG64

6.3. Figures and Tables

7. Evaluation: Differential Diagnosis Performance

7.1. Evaluation methods

7.2. Evaluation results and Discussion

7.2.1. Disease with characteristic symptoms

7.2.2. Difficult-to-diagnose disease with few characteristic symptoms

7.2.3. Case with diagnostic errors

7.2.4. Differential diagnostic performance using the Gumbel Approximate NDCG loss

7.3. Figures and Tables

8. Conclusion

8.1. Evaluation results

8.2. Differential Diagnosis process by physicians and Learning to Rank by machines

8.3. Case data and Information Retrieval

8.4. Potentials for Clinical Decision Support System

8.5. Evolution into Explainable Clinical Decision Support System

8.6. Figures and Tables

9. Abbreviations

10. Declarations

10.1 Ethics approval and consent to participate

10.2 Consent for publication

10.3 Availability of data and materials

10.4 Competing interests

10.5 Funding

10.6 Authors' contributions

10.7 Acknowledgements

11. References

Additional Declarations

Status:

Journal Publication

Version 2