Low light level image enhancement is an important branch of image processing. Its purpose is to highlight the useful features of the image, weaken or eliminate interference information, and make the original unclear or low brightness image clear [1].For example, at night or in rainy or indoor environments, where the light is complex and weak, resulting in a severe decrease in visibility and contrast, the images or videos obtained contain a lot of noise and appear uneven in brightness, which not only affects the subjective perception of the images but also the further processing of these images or videos (e.g. object recognition[2] and detection[3–5]).As a result, shimmer image enhancement has been a hot topic in the field of image processing and has increasingly important research applications. Usually, a good or bad shimmer image enhancement method is assessed by its effectiveness in enhancing the visibility of dark areas in an image, which is usually achieved by increasing the brightness of the dark areas. To achieve satisfactory perceptual quality, it is crucial to enhance the naturalness of the image[6], In particular, the image should not be over-sharpened during the luminance enhancement process. Therefore, to improve image contrast, brightness and quality, scholars have investigated the problem from different angles, such as histogram equalisation (HE) algorithms in traditional methods [7], algorithms based on Retinex model [8] and algorithms based on the defogging model [9].
Histogram equalisation algorithms are used to enhance dark areas by stretching the dynamic range of the image histogram [10], but this algorithm can lead to over-enhancement of the image and loss of image detail. Zuidereld proposed a histogram equalisation algorithm for limiting contrast adaption[11].J.Y. Kim improved the processing speed by overlapping chunks of images followed by histogram equalisation, allowing the images to then retain detail while increasing processing speed[12].Reza.A proposed the adaptive histogram equalisation algorithm, which limits the local image contrast and improves the image over-enhancement phenomenon[13].P.P. Banik converts the image from RGB colour space to HSV colour space and then performs a histogram equalisation of the V channel, which allows for better colour retention[14].However, in essence, they focus on contrast enhancement rather than exploiting the true lighting process, with the risk of over and under enhancement. Another solution is gamma correction, which is a non-linear operation on the image. The main disadvantage is that the non-linear operation of gamma correction is performed on each pixel individually, without regard to the relationship between a particular pixel and its neighbors, and can therefore make the enhancement results fragile and visually inconsistent with the real scene.
Retinex model was proposed by Land in 1971 and is also known as retinal cortex theory. This theory suggests that the image can be decomposed into two components, the illumination component and the reflection component. Land proposed a Retinex algorithm for centre surrounds based on this theory. Later, Jobson proposed the Single Scale Retinex algorithm (SSR algorithm) by combining Gaussian filtering methods[15],Then, a multi-scale retinex algorithm (MSR algorithm) was proposed on the basis of the SSR algorithm [16].However, both SSR and MSR algorithms suffer from colour distortion, so the MSRCR algorithm with color recovery was proposed[17].However, these algorithms use the reflection component as the final enhancement result, which often makes the results look unnatural and often appear to be over-enhanced. Fu proposed a weighted variational model, SRAIE (Simultaneous Reflection and Illumination Estimation), which estimates both the illumination and reflection components simultaneously[18].K Ganga Bhavani proposed the LIME (Shimmering Image Enhancement by Illumination Map Estimation) algorithm, which first uses the RGB channels of the original shimmering image, in three channels, to obtain the maximum value of the illuminated image, and then modifies the original illuminated image continuously by a structural prior to obtain the final illuminated image[19], but the method is less effective in processing images with very low illumination.
K Ganga Bhavani proposed the LIME (Shimmering Image Enhancement by Illumination Map Estimation) algorithm, which first uses the RGB channels of the original shimmering image, in three channels, to obtain the maximum value of the illuminated image, and then modifies the original illuminated image continuously by a structural prior to obtain the final illuminated image [19], but the method is less effective in processing images with very low illumination.
To solve the above problems, this paper proposes an improved light-adaptive enhancement algorithm for shimmering image enhancement. The algorithm consists of two main aspects: the processing of the illuminance component and the reflection component obtained from the Retinex model decomposition, respectively. For the illuminance component, this paper proposes a global luminance adaptive enhancement and a local contrast enhancement algorithm. The global adaptive luminance enhancement method can effectively enhance the luminance of the low-illuminance part, while for the high-illuminance part, the enhancement is relatively small. After a brightness boost, the contrast of an image generally decreases, so this paper applies a local contrast enhancement algorithm to achieve a boost in image contrast and detail. As for the reflection component, since it contains high frequency information and details of the image, this paper proposes a detail enhancement method to deal with the reflection component.
The processed illuminance component and reflection component are reconstructed by Retinex model. By processing the images in the dataset, it is found that the reconstructed enhancement images are grayed out. To address this problem, this paper proposes a non-linear variation method to reprocess the enhancement maps and finally obtain shimmering enhancement maps with better subjective visual quality.
The main contributions of this paper are as follows:
-
For the decomposed illuminance component, this paper proposes a global brightness adaptive enhancement and local contrast enhancement algorithm. While effectively improving the image brightness, the image contrast is also appropriately enhanced, so that the image details are improved.
-
For the obtained reflection component, multi-scale Gaussian blur is used, and then subtracted from the original image to obtain different levels of detail information. Then, the detail information is fused into the reflection component through a certain combination, which makes the texture details of the image more accurate and improves the image quality.
-
For the graying phenomenon of the synthesized enhanced image, this paper proposes a nonlinear change method to further improve the subjective visual quality of the image.
2.PROPOSED METHOD
Retinex model has been proved to be an effective method for low light level image enhancement. How to deal with the light component and reflection component obtained from Retinex model is the key to affect the quality of enhanced image. To better apply Retinex model to achieve more effective low light level image enhancement, this section proposes a Retinex image enhancement method based on light adaptive enhancement.
This method includes the following four steps:
-
First, the original image is converted from RGB space to HSV color space, and then the V component is decomposed into reflection component and brightness component through the retinex model. The brightness component contains the brightness information of the image, and the reflection component contains the details of the image.
-
For the obtained reflection component, multi-scale Gaussian blur is used, and then subtracted from the original image to obtain different levels of detail information. Then, the detail information is fused into the reflection component through a certain combination, which makes the texture details of the image more accurate and improves the image quality.
-
Then multiply the illuminance component and the reflection component and restore them to the V component. The V component will appear graying, so the V component will be nonlinear enhanced.
-
Finally, the final enhanced image is obtained by returning RGB images.
Figure 1 shows the flow chart of the low light level image enhancement method proposed in this paper.
2.1. Retinex model background
Retinex model is based on human visual perception system. The model believes that the image can be decomposed into two parts: illumination and reflection, Namely:
In the formula, refers to a pair of image, , is the illumination component o, ༌which represents the incident light im, ge༌that is the brightness of the light on the object surface in the im, ge༌and determines the dynamic range of the pixel value in the image; is the reflection component of the object surface to the object, which contains the detailed texture information of the image and represents the internal attributes of the image. The basic idea of Retinex model is to remove or reduce the impact of the incident light image through some methods, and retain the original appearance of the image as much as possible.Therefore,the estimation, decomposition and subsequent processing of illumination and reflection components are the key issues of Retinex model. In essence, the illumination component is expressed as the brightness of the object surface in the image, mainly from the smooth areas in the image; The reflection component is generally distributed in the area with small texture in the image. Therefore, Retinex decomposition can be summed up as separating the smooth area and texture area of the image.
2.2 Estimation of illumination and reflection components
To reduce the color bias after image enhancement, this paper converts the original image from RGB space to HSV color space, then only V channel is processed afterwards. First, take the logarithm of the two sides of Eq. (1), then this paper can get the original appearance of the object without considering the nature of the incident light, as follows:
The proposer of Retinex model points out that the illuminance component can be obtained by Gaussian blurring the original image ,which is actually the use of Gaussian filtering to obtain the smooth area of the image. However, Gaussian filtering is not good at image edge processing, and it is easy to lose details. Therefore, this paper use guided filtering to to obtain the illuminance component .Guided filtering is an edge preserving algorithm based on local linear model[21].The effect of edge preserving is better than other filters.
In this paper, similar to the MSR algorithm, multi-scale guided filtering is applied to the original image to extract the illumination component, this paper use three different filter radii to guide filtering processing, and integrate the advantages of different scales to maintain the balance between smoothing and edge details. The formula is as follows:
Where N is the total number of scales. This paper takes N = 3,Represents the weight of the nth scale, that is the weighting factor of the pilot filte,, ,This extract.is a guided filter function.
Estimate the reflection component when the radiation component is obtained. The expression is as follows:
2.3. Enhancement of illumination and reflection components
After processing the V component to get the illumination component of the image. In this paper, a global adaptive brightness adjustment curve is designed based on the characteristics of image illumination distribution, which can suppress high brightness areas, enhance low brightness areas, and process complex and uneven illumination images. The algorithm is expressed as follows:
In the formula,is the illumination component after brightness enhancement,is the original illumination component, and a is the enhancement factor. Figure 2 below shows the enhancement of different brightness values at 2, 3 and 4, respectively.
As you can see from the above image, this method can significantly improve the low brightness, and the higher the value of a, the more significant the low brightness, but it can also cause the dynamic range of brightness to be reduced, resulting in a decrease in the contrast of the image. Based on this, the value a = 3 is selected in this paper. Of course, this paper prefers to enhance the brightness and improve the contrast reduction at the same time. For this reason, this paper improves a local contrast enhancement algorithm [22] to enhance the contrast.
The formula is as follows:
In the formula, is the result of contrast enhancement, and exponent is as follows:
is the convolution of the original illumination component. In document [22], Gaussian filtering is adopted for processing, but the Gaussian filtering will affect the edges. As shown in Fig. 3, Fig. 3 (a) is an enhancement image after Gaussian filtering, it can be seen from the figure that after Gaussian filtering, continuous small black dots will appear at the edges of car wheels and Christmas trees, indicating that there is no enhancement here, and the edge information is lost during processing, this paper use guided filtering for convolution, as shown in Fig. 3 (b), and the edge is improved.
is an image-related parameter, if the contrast of the original image is poo, ,should be a larger value to improve the overall contrast of the image. This paper determine the size of by calculating the global mean square deviation of the original brightness image.
In the formula,is the global mean square deviation of the original illumination. When is less than 3, the image contrast is poor,value should be higher at this time. When the mean square deviation is greater than 10, the original contrast can also reduce the intensity of enhancement. The mean square deviation is between 3 and 10 with an appropriate linear enhancement.
The enhancement effect is shown in Fig. 4.
Because the illumination component is the result of multiscale guided filter processing on the original image, the illumination component is lost in some details, as shown in Figure (b). Figure (c) represents the illumination image after global brightness enhancement. As you can see from the figure, the contrast is further reduced due to the dynamic range compression of the brightness. Figure (d) is an image after local contrast enhancement. From the image, this paper can clearly see that the contrast of the image has been improved, and many details have been reflected.
| Algorithm1: Illumination component enhancement |
| 1: Input༚V Channel Image,Guided Filter Radius r, ,parameter, ༌Mean Variance ; 2: Initial assignment: Guided filter radius r1=[15, 55, 85], a = 3; 3: The illumination component is derived from formula (4); 4: Global adaptive brightness enhancement is applied to the degree component through formula (6); 5: Then the local contrast enhancement of the illumination component after brightness enhancement is performed by the formula (7), (8), (9); 6: Output༚Final enhanced illumination map . |
In Retinex model, the reflection component contains the texture information, i.e., detail, of the image. So for the reflection component, this paper need to enhance its detail. Here, this paper use an image detail enhancement method based on multiscale Gaussian filter [23]. The algorithm is as follows:
Here,andare Gaussian filter functions with standard deviations of and ,respectively.The radius of the Gaussian filter kernel is .Fine detail , medium detail , and coarse detail are calculated using the following formulas.
Hereis the final global detail,andare 0.5, 0.5, and 0.25, respectively. When fine detail is added to the reflective component , the gray difference near the edge is enlarged, but due to its overshoot, the gray level may be saturated. To overcome this problem, in Formula (10),this paper ensure the average difference between the positive and negative components by reducing the positive component in and amplifying the negative component in . Finally, this paper add global detailsto the reflection component to get the detail-enhanced reflection component .
| Algorithm2: Reflection Component Detail Enhancement |
| 1: Input༚Standard deviation of multiscale Gaussian filter α; 2: Initial assignment: Gauss filter standard deviation α=[1, 2, 4]; 3: Get the reflection component from formula (4); 4: Detail enhancement of the reflection component is obtained by the formula (10), (11), (12), (13); 5: Output༚Reflection component for detail enhancement |
Figure 5 shows the impact of multi-scale detail enhancement. In Fig. 5 (a), the visibility of the leaves of the Christmas tree is low. In contrast, in Fig. 5 (b), details enhancement makes the visual perception of local details clearer without any obvious artifacts.
In traditional Retinex model algorithms the extracted reflection components are often used as the final enhanced image, which often results in over-enhanced images with sharp edges. Therefore, in this paper, the enhanced light and reflection components are combined again with the Retinex model. that is:
Here, is a V-channel enhanced image, because this paper deals with the V-channel in HSV color space, so the enhanced V-channel needs to be converted back to RGB color space to get the RGB enhanced image.
2.4 Nonlinear transformation
After obtaining the enhanced image, this paper finds that there is a certain degree of gray flooding in the image. The traditional methods to overcome this phenomenon include linear transformation, piecewise linear transformation and non-linear transformation. Here, this paper use a non-linear change algorithm to process the restored .The following non-linear mathematical expressions are used in this paper:
is the stretched image, is the stretched image, in this paper, the stretched image is the reconstructed enhanced image, and is the average value of extracted.
The effect of the non-linear transformation on the enhanced image is shown in Fig. 6 below. From the picture, this paper can see that after the non-linear transformation, the color of the image starts to be bright and bright, and it is more full.
3.Experiment
In this section, this paper qualitatively and quantitatively evaluates the performance of the proposed Retinex low-light intensity enhancement algorithm based on global adaptive brightness enhancement. The performance of the proposed illuminance reflection double enhancement model is demonstrated by comparing it with existing methods for low-light-level image enhancement, including HE [10], CLAHE [24], SSR [15], MSRCR [17], Dong [25], LIME [19], CRM [26]. All the code runs on Asus notebooks with Intel Core i7-8550U CPU and 32GB memory.
Data: To verify the effectiveness of the algorithm proposed in this paper, this paper have done a lot of contrast experiments on low-light image enhancement. This paper validate the effectiveness of this algorithm by using 35 low-light images (35 images) collected from article [19, 25, 26]and 200 images randomly extracted from Dark Face dataset[27] (hereinafter referred to as 200 Darkface), and 200 images extracted from LOL dataset [28], and their corresponding high-light images.
Objective Indicators: this paper use different objective indicators to evaluate the performance of low-light-level image enhancement. Comparison methods are usually evaluated based on two common criteria: The No Reference Image Quality Evaluation (NIQE) Measuring Natural Image Quality Method [29]. The smaller the NIQE value, the higher the image quality. Because the images in the 35 images and the 200 DarkFace datasets do not have ground truth. this paper can't calculate peak signal-to-noise ratio (PNSR) and structure similarity index (SSIM), so this paper use NIQE for comparison. For the LOL dataset, this paper compare SSIM, PNSR, and mean gradient (MG) on the dataset to achieve a more objective evaluation.
3.1 Illumination component and reflection component enhancement
This paper presents a method for double enhancement of the illumination and reflection components derived from the decomposition of the Retinex model. First, for the illumination component, this paper proposes a global adaptive brightness enhancement and a local contrast enhancement algorithm. For the global adaptive brightness enhancement, this paper set a promotion factor a = 3 to enhance the brightness by formula (6). To reduce the contrast degradation due to brightness enhancement, this paper then proposes a local contrast enhancement algorithm, in which the global mean square deviation is used to measure the contrast of a picture and the formula (7) is used. For the reflection component, this paper uses the formula (11, 12, 13) to enhance the detail.
The Retinex model algorithm assumes that the illumination component is usually smooth, and there is no ground truth for the illumination and reflection components in general, so it is a difficult problem to quantitatively evaluate the decomposition results of the existing Retinex model methods. To evaluate the effectiveness of this method, this paper compare illumination and reflectance with some traditional Retinex models, including SSR[15], MSRCR[17], LIME[19], CRM[26] methods. Similar to the above method, this paper first converts the low light level image from RGB space to HSV space, then decomposes the illuminance and reflection components of the V component according to Retinex model, and then carries out subsequent processing. Finally, the processed dual components are reconstructed back, and then returned to RGB space to obtain the final enhancement image.
Figure 7 is a low light level image selected from the LOL dataset, and then the illuminance and reflection components are estimated using SSR, MSRCR, LIME, CRM and the methods in this paper, as well as the final enhancement image, and then compared. It can be seen from the figure that the illumination and reflection double enhancement method proposed in this paper can effectively improve the main structure information in the illumination component and enhance the detail information in the reflection component. The final result of low light level enhancement is the best.
3.2 Image Enhancement at Low Illumination
this paper run HE, CLAHE, SSR, MSRCR algorithms from traditional algorithms and Dong, LIME, CRM algorithms from the Internet on 35 images and 200 Darkface datasets, respectively. Table 1 compares the average NIQE values of these methods in the two datasets and shows that our method is better than others. Figure 8 and Fig. 9 show the contrast of enhancement results obtained by different low light level enhancement methods for low light level images selected from 35images and 200 Darkface datasets.
Table 1
Average NIQE results for 30 images and 200 images from darkface datasets by different methods.
| Dataest | 30images | 200darkface |
| Metric | NIQE | NIQE |
| HE | 3.65884 | 2.60318 |
| CALHE | 3.61642 | 2.55674 |
| SSR | 4.35074 | 2.99542 |
| MSRCR | 4.0529 | 3.03472 |
| Dong | 3.03722 | 2.48356 |
| LIME | 3.75358 | 2.70876 |
| CRM | 3.65002 | 2.83018 |
| our | 2.95424 | 2.35988 |
| From these comparisons, this paper can see that the result image enhanced by this method has better subjective visual quality than other methods. Traditional methods such as HE, CLAHE, MSRCR will cause problems such as over-enhancement or color distortion, while Dong method will have problems such as black edges. As shown in Table 1, the objective evaluation results of the proposed methods are obviously better in NIQE. |
| This paper also implements the method proposed in this paper on LOL dataset to verify the effectiveness of extreme dark image enhancement; Figs. 10 and 11 show two extremely dim and weak light images selected from the LOL dataset, the corresponding ground truth image, and the images enhanced by different methods; It can be seen from Fig. 10 and Fig. 11 that the enhanced image of the proposed method is closer to the ground truth image, the brightness of the enhanced image obtained by HE and CLAHE methods has not improved much, the enhanced image obtained by SSR and MSRCR methods has obvious color deviation, and the result image obtained by LIME method has excessive enhancement; Compared with other methods, the method proposed in this paper has better enhancement effect and higher quality of enhanced image. This method realizes chromatic aberration correction and effectively improves the subjective perception of the image. |
This paper selects 200 images from the LOL dataset for processing, calculates the SSIM and PNSR results of 200 images respectively, and then takes the average value. The final results are shown in Table 2. It can be seen that the results of the proposed method in SSIM and PNSR are obviously better than those of other methods.
Table 2
Quantitative comparison of LOL datasets in PNSR and SSIM.
| Metries | HE | CALHE | SSR | MSRCR |
| PSNR | 12.301 | 14.246 | 15.641 | 15.725 |
| SSIM | 0.448 | 0.594 | 0.662 | 0.682 |
| Metries | Dong | LIME | CRM | Ours |
| PSNR | 16.317 | 16.599 | 16.574 | 17.741 |
| SSIM | 0.697 | 0.664 | 0.659 | 0.765 |
Table 3 compares different image enhancement methods at low light 1080 × Runtime on 720 images. Runtime is the average of time calculated from 20 low-illumination images. this paper note that our method is not faster than all other methods; this paper will further improve our algorithms and procedures in future work to improve the speed of operation.
Table 3
Comparing the calculation time (s) of different low-light intensity image enhancement methods.
| Method | HE | CALHE | SSR | MSRCR |
| Time | 1.21 | 2.78 | 1.48 | 2.07 |
| Method | Dong | LIME | CRM | Ours |
| Time | 1.56 | 6.73 | 3.41 | 2.56 |
3.3 Low light level night vision image enhancement
This paper also processes the low light level image obtained from the low light level night vision instrument to further verify the universality of the method proposed in this paper[30]. The results are as shown in Figs. 12 and 13. It can be seen from Figs. 12 and 13 that the HE and CLAHE methods have poor effect in details promotion, and many details are lost; SSR and MSERCR methods cause color distortion and loss of details; The Dong and CRM methods are over enhanced, which reduces the image contrast; LIME also suffers from loss of detail and contrast loss. The proposed method is better than other methods in details retention and contrast enhancement.
4.Conclusion
Retinex model has been proved to be an effective method for low light level image enhancement, and how to deal with the illuminance and reflection components decomposed from Retinex model is the key to obtain high-quality enhanced images. In order to better apply Retinex model to achieve more effective low light level image enhancement, this paper proposes an image enhancement method based on Retinex model with dual enhancement of illumination and reflection. Firstly, the illumination component and reflection component are obtained by Retinex model decomposition based on guided filtering, and then the illumination component is enhanced by global adaptive brightness enhancement and local contrast enhancement; The reflection component is enhanced in detail. Finally, the enhanced illumination component and reflection component are combined into an enhanced image. The effectiveness of this method is proved by a large number of low light level image enhancement experiments, and it has certain advantages compared with other low light level image enhancement methods.