H-type indices with applications in chemometrics：h accuracy index (HAI) for evaluating and comparing errors in analytical chemistry

doi:10.21203/rs.3.rs-2969817/v1

Inspired by the popular h-index in bibliometrics, an h-accuracy index (HAI) was suggested as a new indicator to evaluate and compare errors in analytical chemistry. HAI simultaneously considers the ‘trueness’ of analytical measurements and the frequency of measurements with high ‘trueness’. HAI was defined as: if the ‘trueness’ of at most M% of the total measurements is no less than M%, the value of HAI will be M%, where a specific definition of ‘trueness’ was given to compute HAI. The range of HAI was between 0 and 1. HAI was used in two problems: (1) as a new index of errors, to evaluate and compare the analytical results by different analytical methods; and (2) to evaluate and compare the prediction performances of different multivariate calibration models. Data analysis indicated that HAI was a reasonable, robust, easy-to-compute, and comprehensive index for evaluating and comparing errors in analytical chemistry.

Analytical Chemistry

Applied Statistics

h-index

h-accuracy index (HAI)

analytical errors

multivariate calibration

method evaluation

Academic influence is an essential embodiment of a scholar, an institution, or a country or region's contribution to science and technology [1]. Moreover, academic influence is often accompanied by the incentive mechanism of scholars and institutions, which will in turn play a key role in planning and development of scientific research [2]. Therefore, how to promote the evaluation of academic influence to be more objective and reasonable has always been one of the hot research topics in the field of library and bibliometrics [3, 4].

Due to the complexity in evaluation of scientific research outputs, traditional criteria are not satisfactory [5, 6] and many scholars devoted themselves to the research of better evaluation indices for academic influence [7–9]. In 2005, Professor Hirsch, a physicist at the University of California, San Diego, creatively proposed a new index, h index [10], to measure the academic influence of scholars. It not only considers the citations of papers published by a scholar, but also considers the number of highly cited papers. Moreover, the calculation method is simple and reasonable. It has aroused strong interest in the whole scientific community in evaluating academic influence [11, 12] and has been regarded as a significant contribution to bibliometrics. Its application soon expanded to evaluating the academic influence of journals, institution, patents, and funds [13–16]. Moreover, many improved versions of h-index and related indices have been proposed and studied [17–19].

As a data analysis method, how can h index achieved the great popularity and success in bibliometrics? The following features of h-index may shed light on the above question and might also be interesting to chemometrics:

(1) it is very simple and easy to compute;

(2) it combines more than one aspects of the data, e.g., the number of citations (strength of a quantity) and the number of highly cited papers (the coverage of high quantity across objects), in a single indicator;

(3) it is not simply an accumulation or average of the data but it is robust against a few outliers.

Inspired by idea of the original h-index, in this work, an h-accuracy index (HAI) was suggested to evaluate and compare the errors of different analytical methods or multivariate calibration models. HAI was compared with some traditional analytical indices to demonstrate its characteristics and advantages.

2.1. h-index in bibliometrics

In bibliometrics, the definition of h-index was:

A scientist has index h if h of his or her N _p papers have at least h citations each and the other (N_p − h) papers have ≤ h citations each.

In Fig. 1, the citations of the 20 papers are arranged in a descending order and 10 of the 20 papers has at least 10 citations; while each of the other 10 papers has no more than 10 citations. As a result, the h-index will be 10 in this example. Geometrically, h-index is the width of the largest square under the curve of citations.

[Figure 1]

2.2. h-accuracy index in analytical chemistry

The h-accuracy index (HAI) was defined as:

For N analytical measurements, if at most M% of the N measurements have a ‘trueness’ no less than M%, the h-accuracy index (HAI) of the analysis will be M%.

In this work, a specific ‘trueness’ (T) of a single measurement was defined as:

$${T}_{i}=\text{m}\text{a}\text{x}\left\{0, 1-\left|\frac{{x}_{i}-x}{x}\right|\right\}$$

where x_i is the value of the ith measurement and x the reference or mean value.

Some remarks can be made on HAI. Just like the original h index could measure the overall citations of several papers, HAI will measure the overall ‘trueness’ of a series of measurements in the same manner with a single index. HAI simultaneously considers the ‘trueness’ of measurements and the frequency (in percent) of measurements with high ‘trueness’, therefore, it might be a reasonable index to reflect the analytical quality or capacity. As for the value of ‘trueness’, obviously, the closer a measurement is to the reference or mean value, the higher the ‘trueness’ value is.

[Figure 2]

To exemplify the computation of HAI, suppose one has taken 10 measurements and 9 of them (90%) have a ‘trueness’ value no less than 0.90, the HAI of the analysis will be 0.9 (Fig. 2).

2.3 Software

The data analysis was performed using MATLAB R2013b (Mathworks, Sherborn, MA, USA). The MATLAB codes to compute HAI can be requested from the corresponding authors.

3.1. To evaluate and compare the quality of different analytical methods

The analytical results of two analytical methods for 15 test objects with reference values (Table 1) were simulated and used to compare different analytical indices. For the analytical results in Table 1, the average absolute relative deviation (AARD) of the two methods were 1.4% and 5.0%, respectively; and the root mean squared error (RMSE) of the two methods were 0.11 and 0.35, respectively. Both AARD and RMSE indicated that the method 1 is more accurate and has lower analytical errors than the latter. The HAI of the two methods were 0.991 and 0.970, respectively. For the method 1, an HAI value of 0.991 means that at most 99.1% of all the measurements’ ‘trueness’ was no less than 0.991. Obviously, a higher HAI corresponds to a more accurate analytical method. In this example, HAI could obtain the similar results as RMSE and AARD and seemed to be a reasonable index to evaluate an analytical method.

Table 1

Simulated results obtained by two analytical methods
Samples	Reference value	Assay value by method 1	Assay value by method 2
1	8.17	8.12	8.74
2	9.34	9.33	9.18
3	2.352	2.303	2.159
4	5.576	5.758	5.397
5	16.42	16.33	16.13
6	2.459	2.415	2.161
7	18.45	18.29	18.14
8	13.47	13.46	13.19
9	12.50	12.60	12.25
10	8.50	8.40	9.21
11	7.989	7.796	7.628
12	2.947	3.087	2.509
13	19.16	19.17	19.32
14	6.564	6.660	6.303
15	14.44	14.41	14.63

Table 2

Simulated analytical results of one object with 10 repeats
	10 repeated measurements ^a
Method 1	9.94	9.92	10.05
	9.82	9.66	10.10
	9.98	9.89	10.16
	10.09	10.71
Method 2	9.96	9.99	9.85
	10.12	10.11	10.34
	10.17	10.06	10.15
	9.95	10.36
^a The reference assay value was 10.00.

[Table 1]

[Table 2]

Moreover, in Table 2, two analytical results were simulated with 11 repeated measurements of an object with the reference value of 10.00. Seen from the boxplots (Fig. 3), the two methods seemed to give similar and comparable results except that the method 1 had an obvious outlier among the 11 measurements. With the outlier, the relative standard deviation (RSD) and RMSE of the two methods were 0.26 vs 0.18 (RMSE) and 2.7% vs 1.6% (RSD), respectively. Obviously, the outlier can severely influence the values of RMSE and RSD. In comparison, the HAI values of the two methods were 0.990 vs 0.988, indicating the results of the two methods were similar and comparable. Because the mean value of the method 1 was more close to the reference value than that of the method 2, the method 1 even had a slightly higher HAI value. By removing the 11th measurements of the two methods, which had the largest deviations from the reference value, the RSD and RMSE of the two methods became 0.15 vs 0.15 (RMSE) and 1.5% vs 1.4% (RSD), respectively. The HAI values of the two methods were 0.991 vs 0.989, which was almost the same with that obtained by including the outliers. With the outliers removed, RMSE, RSD and HAI gave the similar evaluation results of the two methods. It was demonstrated by the above data analysis that HAI was a more robust index than RSD or RMSE.

[Figure 3]

3.2. As a new index to evaluate multivariate calibration models

In multivariate calibration, the root mean square error (RMSEP) [20, 21] and the predictive squared correlation coefficient (Q²) [22, 23] are two usually used indices to evaluate and compare model prediction ability. In this work, HAI was compared with RMSE and Q² to evaluate model prediction performances of multivariate calibration models including different number of latent variables (LVs). This meat data set [24] was used for comparison of different indices. The near-infrared transmission (NIT) spectra in the wavelength range of 850–1050 nm (100 channels) were used to calibrate the fat content of finely chopped pure meat. The data set had 172 training objects and 43 test objects.

A previous study [25] indicated that PLS models with 14 LVs would be suitable for multivariate calibration. In this work, the RMSEP, Q², and HAI values of 4 PLS models obtained by including 12–15 LVs were computed and compared (Table 3). Seen from Table 3, RMSEP and Q² gave consistent results of model comparison, by including more LVs, the model had a lower RMSEP and higher Q². The results of HAI were slightly different. The HAI values of different PLS models were 0.923 (12 LVs), 0.919 (13 LVs), 0.928(14 LVs), and 0.916 (15 LVs), respectively. Why did PLS models using 13 LVs and 15 LVs give slightly lower HAI values? This could be explained by the definition of HAI and ‘trueness’, where deviations of the lower reference values tend to give a lower ‘trueness’ and HAI. The predicted values vs reference values were plotted in Fig. 4, where the PLS models with both 13 LVs and 15 LVs seemed to have more deviations of lower reference values (marked in cycles). Generally, the prediction results by using 12–15 LVs are generally fine and HAI could give similar evaluations as RMSEP and Q². Different from RMSEP and Q², which are computed by accumulations and means of all the deviations, HAI simultaneously considers the deviations and the number of measurements with lower deviations. As discussed in the previous example, HAI is also a robust index that is less sensitive with a few larger deviations. The results indicated that HAI could provide a reasonable and useful alternative index to the traditional RMSEP and Q² for evaluating multivariate calibration models.

Table 3

Prediction results of fat content in meat by PLS including 12–15 LVs
	RMSEP	Q²	HAI
PLS with 12 LVs	2.24	0.971	0.923
PLS with 13 LVs	2.10	0.974	0.919
PLS with 14 LVs	2.01	0.976	0.928
PLS with 15 LVs	1.97	0.977	0.916

In this work, inspired by the popular h-index, an h-accuracy index (HAI) was proposed as a new index of analytical errors. HAI was used to evaluate and compare the results of different analytical methods and multivariate calibration models. Compared with the traditional indices, including AARD, RSD, RMSE, RMSEP, and Q2, HAI could give the similar results and proved to be a reasonable analytical index. Moreover, HAI simultaneously considers the low deviations and the number of measurements with lower deviations and is robust against a few higher deviations.

ACKNOWLEDGEMENTS

Authors are grateful to the financial support from the National Natural Science Foundation of China (No. 21803009), Guizhou Provincial Science and Technology Department (Nos. QKHJC[2017]1186, QKHZC[2019]2816, QKHZC[2019]2383, and QKHPTRC[2020]5009), Guizhou Provincial Department of Education (No. QJHKYZ[2018]073), Tongren Science and Technology Bureau (No. TSKY2019-3), the Science Foundation of Educational Commission of Hubei Province of China (No. T2020008), Hunan Provincial Department of Education (No. 16A236), and Shaoyang University (No. CX2019SY052).

Author contribution

Lu Xu: Conceptualization; Methodology; Writing; Software.

Qin Yang: Software; Writing; Original Draft;

Xihui Bian: Data Curation; Formal analysis; Review & Editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Zhang F, Wu SL. Predicting future influence of papers, researchers, and venues in a dynamic academic network. J Informetr. 2020;14(2):101035.
Ariel Xu Q, Chang V. Co-authorship network and the correlation with academic performance. Internet of Things 2020;12:100307.
Davis FM, Obi AT, Gallagher KA, Henke PK. Accessing the academic influence of vascular surgeons within the National Institutes of Health iCite database. J Vasc Surg. 2020;71(5):1741-1748.e2.
Giatsidis C, Nikolentzos G, Zhang CH, Tang J, Vazirgiannis M. Rooted citation graphs density metrics for research papers influence evaluation. J Informetr. 2019;13(2):757-768.
Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295(1):90-94.
Hirsch JE. Does the h index have predictive power? Proc Natl Acad Sci USA 2007;104(49):19193-19198.
Ball P. Index aims for fair ranking of scientists. Nature 2005;436:900.
Kulkarni AV, Aziz B, Shams I, Busse JW. Comparison of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA 2009;302:1092-1096.
Winter JCFD, Zadpoor AA, Dodou D. The expansion of Google Scholar versus Web of Science: a longitudinal study. Scientometrics 2014;98:1547-1565.
Hirsch JE. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA 2005;102(46):16569-16572.
Kreiner G. The Slavery of the h-index-Measuring the Unmeasurable. Front Hum Neurosci. 2016;10:556.
Root Kustritz MV, Nault AJ. Measuring productivity and impact of veterinary education-related research at the institutional and individual levels using the H-Index. J Vet Med Educ. 2019;47(4):e0618072r1.
Ernat JJ, Yheulon CG, Lopez AJ, Warth LC. Does the h-index and self-citation affect external funding of orthopedic surgery research? An analysis of fellowship directors and their subspecialties. J Orthop. 2020;20:92-96.
Kuan CH, Huang MH, Chen DZ. Ranking patent assignee performance by h-index and shape descriptors. J Informetr. 2011;5(2):303-312.
Primo de Carvalho Alves L, Rodrigues AA, Moreira Lima F, Janovik N, Sanguinetti Czepielewski L, Mendes Ceresér KM, Sica da Rocha N. H-index may influence more than methodological variables for publication in high impact psychiatry journals – A systematic review. Eur Psychiat. 2017;41:S382.
Ramos MB, Teixeira MJ, Preul MC, Spetzler RF, Figueiredo EG. A bibliometric study of the most cited reports in central nervous system arteriovenous malformations. World Neurosurg. 2019;129:261-268.
Aoun SG, Bendok BR, Rahme RJ, Dacey RG, Batjer HH. Standardizing the evaluation of scientific and academic performance in neurosurgery--critical review of the "h" index and its variants. World Neurosurg. 2013;80(5):e85-e90.
Franceschini F, Maisano D. Structured evaluation of the scientific output of academic research groups by recent h-based indicators. J Informetr. 2011;5(1):64-74.
Jenkins NR. Variation in the h-Index and its use in the assessment of academic output. World Neurosurg. 2016;87:619-620.
Faber NM. Estimating the uncertainty in estimates of root mean square error of prediction: application to determining the size of an adequate test set in multivariate calibration. Chemom Intell Lab Syst. 1999;49(1):79-89.
Ferreira RJ, Rosa TR, Ribeiro J, Barthus RC. Simultaneous metal determination in artisanal cachaça by using voltammetry and multivariate calibration. Food Chem. 2020;314:126126.
Goodarzi M, Sharma S, Ramon H, Saeys W. Multivariate calibration of NIR spectroscopic sensors for continuous glucose monitoring. TrAC Trend Anal Chem. 2015;67:147-158.
Gramatica P, Sangion A. A historical excursus on the statistical validation parameters for QSAR models: A clarification concerning metrics and terminology. J Chem Inf Model 2016;56(6):1127-1131.
Borggaard C, Thodberg HH. Optimal minimal neural interpretation of spectra. Anal Chem. 1992;64(5):545-551.
Xu L, Hu O, Guo YW, Zhang MQ, Lu DW, Cai CB, Xie SP, Goodarzi M, Fu HY, She YB. Representative splitting cross validation. Chemom Intell Lab Syst. 2018;183:29-35.

H-type indices with applications in chemometrics：h accuracy index (HAI) for evaluating and comparing errors in analytical chemistry

Status:

Version 1

Abstract

Figures

1. INTRODUCTION

2. METHODS

2.1. h-index in bibliometrics

2.2. h-accuracy index in analytical chemistry

2.3 Software

3. DATA ANALYSIS AND DISCUSSIONS

3.1. To evaluate and compare the quality of different analytical methods

3.2. As a new index to evaluate multivariate calibration models

4. CONCLUSIONS

Declarations

References

Status:

Version 1