Reducing the number of measuring points of the LED-based colorimetric probe

In this paper, reducing the number of necessary measuring points for estimating a reflected electromagnetic spectrum of a printed color patch is presented. In our previous work, a machine learning-based method was proven to be superior to Cubic Hermite interpolation in estimating spectrum based on six measured values provided by measuring reflection of six LED sources (400 nm, 457 nm, 517 nm, 572 nm, 632 nm, and 700 nm). Now, the new hypothesis is that the number of measuring points LEDs could be decreased without the significant loss of the spectrum estimation. The ECI2002 test chart was used to create the dataset, which was further divided into training and test subset. For all the colors on the test chart, the measurements were performed on printed patches with the device proposed in our previous work, as well as with the commercial spectrophotometer X-Rite i1 Publish Pro2, which were then used as the ground truth, or reference values. The Artificial Neural Networks were trained to estimate spectrums based on measurements acquired with our device. The results proved satisfactory even when the number of measuring points is reduced from six to three RGB LEDs (457 nm, 517 nm, and 632 nm).


Introduction
As the world of Internet of Things (IoT), and sensor-data gathering is becoming more widespread, reducing the cost of each sensor system is becoming an important factor. In our previous work (Batinic et al. 2021) the novel way of estimating the reflected 585 Page 2 of 11 spectrum of a printed sample was presented. The good quality of the estimated reflected spectrum of digitaly printed samples was achieved. One of the drawbacks of the device presented in the previous paper is that, while several of the light-emitting diodes (LEDs) are widely available (red, green, and blue), other are not that easy to acquire, like the LED with a central wavelength at 572 nm. Also, designing the colorimetric probe with six LEDs is complicated. Reduced number of LEDs greatly simplifies mechanical design. In this paper, we are focusing on reducing the number of needed diodes to only Red, Green, and Blue (RGB) LEDs (457 nm, 517 nm, and 632 nm), while still maintaining good results.
Color estimation methods by using reflected signals of RGB LEDs exist, but do not estimate the whole spectrum. (Yang et al. 2007) propose the method of using the RGB LEDs (463 nm, 514 nm, and 628 nm) and then multipoint calibration. Using the professional spectrophotometer PR650 by Photo Research Inc, the estimations were improved significantly after calibration. The method by (Saracoglu and Altural (2010)) is similar but uses the Artificial Neural Network (ANN) instead of multipoint calibration. However, both of the solutions do not measure spectrums, but rather the International Commission on Illumination (CIE) XYZ tristimulus values. Our solution estimates the whole visible spectrum for the printed color patches. This method presents a trade-off between the quality of the spectrum estimation and the low cost of a device as well as the simpler mechanical construction. The proposed device provides better color estimation than colorimeters, while still having much simpler construction and lower cost than the commertial spectophotometers.
Although the main application of the presented device is in the digital print sample spectrum estimation, the need for reduction of the cost could be found in the other industries as well. The paper by (O'Farrell et al. 2006) describes the practical usage of the expensive Ocean Optics S2000 spectrometer. In the large-scale industry of food production, the reflected spectrums of the food could be monitored online, and cooking could be fine-tuned. It is one example of a demand for such a device that could be easily produced and at a low cost, while still providing good quality estimations of the color spectrum.
A similar problem of reducing the number of measuring points, while maintaining good results, is presented in a paper by (Ciosek et al. 2004). The sensor array was used to classify the brands of water and juice. The initial network used a 17-sensor array, and the authors tried to reduce the number of sensors. In the end, they concluded that it was possible to get satisfactory results with only 9 sensors. In the (Winquist et al. 1993), the authors tried to reduce the number of sensors of an electronic nose. The results were partially good, the prediction of a type of meat stayed on the same level, but the estimation of storage time significantly worsened. It was apparent that some sensors are more important than others in case of estimation of storage time. Some important ones for that functionality were removed. The sensors that are removed should be carefully picked because not all of the sensors carry the same amount of information.
The Artificial Neural Networks (ANNs) are great approximators of non-linear functions, and there is a possibility that some of the measured values are less important, so the ANN would be able to estimate spectrum without them. The goal is to try to make a device that will be able to estimate the whole spectrum in the visible range using the fewest measuring points.
The CIEDE2000 metric is going to be used to measure the quality of the estimated spectrums, as it has a strong agreement with human perception of the colors (Ortiza Jaramillo et al. 2016).

Methodology
All of the test samples for color patches are digitaly printed and the reflected values are manually measured. The fiber-optic sensor system presented in our previous papers consists of a signal processing block with microcontroller unit (MCU), six transmiting LED sources, broadband photodetector and colorimetric probe that is connected to device using plastic optical fibers. The block scheme is shown in Fig. 1 a). The measurements are taken using time and frequency multiplexing. In the first half of the period, the three LED sources are active, triggered simultaneously with three different frequencies, and in the second half the other three LED sources are active, triggered with the same frequencies. Broadband Silicon photodetector (TSL14S) with integrated transimpedance amplifier receives the signal and digital filters are used to demultiplex respective responses and get the intensity of reflected signal for the particular source. There are a total of six measuring points, using LEDs of different wavelengths, as shown in Fig. 1 b). The central wavelength of the LEDs are at 400 nm, 457 nm, 517 nm, 572 nm, 632 nm, and 700 nm with respective spectral width of 13 nm, 20 nm, 28 nm, 16 nm, 19 nm and 21 nm. All of the LEDs in this work are packed in through-hole round packages, 5 mm in diameter, with the angle of viewing of 30 • and have ultrabright intensity.
The reference values for training and testing are provided by spectrum measures provided by commercial spectrophotometer X-Rite i1 Publish Pro2, which gives the resulted spectrum in equidistant 36 points. The idea is to use a subset of measured values and train an Artificial neural network (ANN) to estimate a spectrum based on our measurements, the goal is to determine how much the reduction of the number of input values affects the estimated spectrum.
Proposed combinations, besides using all of the measuring points: 1. Five measuring points, without the diode on 400 nm 2. Four measuring points, without the diodes on 400 and 700 nm 3. Three measuring points acquired by using RGB LEDs (457 nm, 517 nm, and 632 nm). For each combination, the ANN would be trained to estimate spectrum using specific measured values for inputs and ground truth from a commercial spectrophotometer as output. The quality of the neural network output spectrum would be estimated using △E 00 (CIE2000) metric.

Analysis of the dataset values
To show the complexity of the problem, and the challenge of approximating the function for spectrum estimation based on measured points, a few representative examples from the dataset will be analyzed. In Fig. 2, the input values (measurement points) are presented with dots, while the reference spectrum is presented with dashed lines.
From Fig. 2 it can be observed that, for saturated colors, measured values follow the spectrum closely. For the dark colors, on the other hand, there are some discrepancies. The main reason for that is that the intensity of reflected light for dark colors is low, so the measurement errors are more pronounced. The most notable difference is with the first measurement point (400 nm) when the colors are darker, the first diode does not carry almost any information. For each of the printed color patches, the measurement is done manually with both EyeOne Pro and our device. Each measurement is susceptible to the imperfect placing of instruments or similar human error, finite sensitivity of sensing element, and errors due to quantization. Also, for the colors close to white, there are some small discrepancies that could lower the approximation quality. However, if the discrepancy is consistent, the Artificial neural network should be able to compensate for it. The importance of each measuring point is also taken into consideration. Since the main application of our device is approximation of the spectrums of 4-colour process printing samples, the source of the colors of the given samples are provided by the combination of standard Cyan, Magenta, Yellow and black (CMYK ) inks. For this particular application some of the diodes carry more information than the others. From the graphs in Fig. 2 it is evident that the last measurement point (700 nm) could be less important than the fourth (572 nm) measuring point and that the last two measuring points could be redundant. Further, the human eye has low sensitivity at 700nm, therefore the end result is less affected by it. If the device would be used for other application, where the source of the color is different, analysis about the relations of the measured values should be repeated.

Dividing the dataset
The ECI2002 chart ISO 12642-2:2006ISO 12642-2: (2006, provided in Fig. 3 is divided into several groups. For example, the first group, which covers rows 1-27 and columns 1-27, contains only combinations of CMY colors, without any black. Then there are groups that add a certain amount of black and make patches in that group darker. The group that covers rows 1-8 and columns 38-45 is printed with 80% black and different CMY combinations, while the group rows 9-28, columns 42-45 show different saturation of pure CMYK colors.
In order to test the trained network properly, the test patches should cover all of the distinct groups equally. On the other hand, there should be enough data for the ANN to learn Fig. 3 Test chart ECI2002 with marked color patches. The patches that are marked with dotted frames are the test samples. Ones that are marked with red X are considered measurement errors. The rest of the samples are used for training of the ANN properly. Therefore, the process of test sample selection was carefully conducted with the intention of having around 20% of the samples for the testing purposes and the 80% to be used in the training process.
Some of the values (marked with red X in Fig. 3) are considered to be measurement errors. Upon finishing the training for the first time, with the whole dataset, some samples had extremely high △E 00 values. When comparing measured points and the spectrum of each of these samples it was evident that some of the measured values are significantly different than the measured values for similar colors/spectrums. Most probable, the probe was not placed ideally and that introduced the random error. The ANN can not compensate for those errors, and they introduce significant problems for the training of the network. Around 1.5% of the colors are removed completely. After removing measurement error samples, there is a total number of 1463 samples left in the dataset. The number of test samples is 277 or around 19%, while the rest (1187) is used for training purposes.

Choosing model and training of the Artificial Neural Network
The essence of this paper is to approximate the 4 functions using machine-learning techniques. The input of those functions is a combination of the amount of reflected light measurements on certain wavelengths (6, 5, 4, or 3 measurement points), and the output is the spectrum estimation in 36 equidistant wavelengths in the visible spectrum range. The paper by (Hornik et al. 1989) establishes that multilayer feedforward networks are universal function approximators. Authors in this paper use that feature of multilayer feedforward networks to approximate output spectrum (36 values), based on the measurements of 6, 5, 4, or 3 values. The simple multilayer feedforward network, Multilayer Perceptron (MLP) (Haykin 1999), is used to approximate the needed functions. The MLP was trained on the training data, and the quality of estimated spectrums was calculated using △E 00 metrics on the test data, which is not used during training.
The size of the MLP greatly influences the quality of the estimated spectrums, but also the time that is needed to train such a network. The larger the network, the faster MLP converges, but there is a possibility that the network would overfit (Haugen and Kvaal 1998). If the network is smaller, it learns at a slower pace, needs more time to converge, but the overfitting of the network is less likely to happen.
The training of the MLPs of the different sizes provided insight into the optimal number of layers and their sizes.
The model of the Fully Connected Neural Network that has an input layer of 3-6 input floating-point values, a hidden layer of 72 neurons, and an output layer of 36 floating-point numbers, turned out to be the optimal configuration.
If the additional layer is added, the loss function would drop significantly and at the fast pace during the training. The estimated spectrums on the test data would, after initial improvement, begin to get worse, as the large network overfits as the training continues. Even the best results with more than one layer turned out to be not as good as with the smaller network, that contains only one layer.
The network with one hidden layer of 72 neurons converges to good results, but at a slower pace than the larger network. Results on the test set do not get worse upon further training, showing no signs of overfitting.
The MLP smaller than 72 neurons in one hidden layer could not provide satisfactory results.

Page 7 of 11 585
The activation function that provided the best result is the Sigmoidal activation function. All of the input and output values are in the range 0-1, so the sigmoidal nonlinear characteristics provide a good foundation for the approximation of needed nonlinear functions. The loss function used for all of the networks is the Mean Squared Error loss function. The framework used for all of the training and testing was Tensorflow. The training of the neural network was handled by using the Nadam optimizer where the learning rate was gradually lowered after a certain number of training epochs. The larger learning rate at the start provides faster convergence times, and the lowering of the learning rate, while the function approaches minima, ensures better results. After a series of experiments the learning rates that were used are as follows: the initial learning rate is set to 0.03, after some time the learning rate is set to 0.01 (the default value of learning rate in Tensorflow/Keras), and lastly, the learning rate is set to 0.005, at which point, after a small amount of time, the results are slowly oscillating around results provided in this paper, there are no significant improvements, even though the loss function is getting smaller. The further tries to reduce the learning rate did not provide any significant improvement in the results.
All of the training configurations were trained several times, in order to avoid getting stuck in local minima, and the results were mostly similar, with minor differences in results. For each configuration, the best results were presented in this paper.

Results and discussion
In this chapter, the results of calculating △E 00 of estimated spectrums for over 277 test samples will be presented in several different ways in order to show how reducing the number of measurement points affects the quality of spectrum estimations. From Table 1 it is evident that the greatest change is with the maximal value of △E 00 , while the mean, median and minimal values are not affected significantly.
The nature of the change is most apparent in histograms that are presented in Fig. 4. and the △E 00 maps for 3 and 6 diodes in Fig. 5. The number of samples with △E 00 > 3 rises as the number of measuring points is reduced. If observed closely, in Fig. 5, it is evident that the majority of the colors that have △E 00 > 3 are dark colors. It can be seen that for the median and mean values, the number of measurement points is not of great importance. Therefore, for most test samples in the test set, the reduction of the number of measuring points is not detrimental. Some of the test samples are actually better with fewer measurement points.
Finally, a few examples of the estimated spectrums are going to be analyzed in more detail. In Fig. 6 the measurement points are presented with small triangles. Each of the combinations of diodes is distinguished by color. For example, the Red triangles represent measurements for the 3 diodes, and the Red dashed line presents the estimated spectrum. The gray line is a referent spectrum. The most common is that the value of △ E 00 stays around 1. In Fig. 6 graph 1, that particular case is shown. In Fig. 6 graph 2, the drastic worsening of the estimated spectrum is apparent when the number of measurement points is lowered from 6 to 3. The most interesting, unique, and unexpected case is shown in Fig. 6 graph 3. For the particular blue color, the estimation is better with fewer measurement points. The explanation for this could be that the measurements with the diode on 400 nm are not that reliable, so it introduces non-systematic error. The last graph in Fig. 6 shows estimated spectrums of all of the combinations. The color is black, so the measuring errors are becoming more important, which needs to be taken into consideration. The quality of all of the spectrums is comparable, and the number of measurement points does not influence the △E 00 significantly.
In electrophotography and ink-jet printing, the reproduction accuracy is considered satisfactory if the color difference of 5 △E 00 is not exceeded (IDEAlliance 2017 [Online]). In graphic industry and offset-printing, tolerances for satisfactory color reproduction are 3.5 △E 00 for Cyan, Magenta, Yellow, and 5 △E 00 for black (ISO 12647-2: Comparing the map of △E 00 for 6 and 3 diodes. This graph corresponds to the test samples from Fig. 3. The gray squares are the training samples from the dataset, and are not being used for testing purpose. The color of the other squares corresponds to △E 00 value for each sample. Gradient from 0 △E 00 (green) to 5 △E 00 (red) 2013). With that in mind, this paper confirms that even with only 3 measuring points (LEDs on 457 nm, 512 nm and 632 nm), the results are satisfactory.

Conclusion
This paper confirms that our device can provide satisfactory spectrum approximations for the samples that are printed using 4-colour process printing, even with only 3 measuring points (LEDs on 457 nm, 512 nm and 632 nm). The main advantage over other colorimeters that use RGB diodes is that our device can estimate the whole visible spectrum, while similar devices can estimate only XYZ tristimulus values.
The greatest challenge proved to be the estimation of the spectrum of the darker colors. The measurement errors are also more prominent for smaller reflection values. As the number of measurement points decreases, it is harder to compensate for those errors, and we get significantly worse △E 00 for dark colors. For the light and saturated CMYK colors, there is almost no difference if the spectrum is estimated using 3 measurement points or all 6. All of the spectrums are around 1 △E 00 . There is always the possibility that the more balanced dataset would improve the results of the dark color estimations. Another technique that could be utilized is Principal component analysis (Jolliffe and Cadima 2016) on the input values of the dataset. A more analytic approach could provide a better understanding of what measuring points to remove, and possibly provide even better results. In addition to reducing the cost of the optoelectronic system, by using only widely available Red, Green and Blue LEDs, the reduction of the number of LED sources significantly simplifies the mechanical design of the colorimetric probe. Quicker time multiplexing of a smaller number of the transmitting signals, and also the decoding on the receiving side, increases the response time of the system.