Captured rainy images severely degrade outdoor vision systems performance, such as semi-autonomous or autonomous driving systems and video surveillance systems. Consequently, removing heavy and complex rain streaks i.e., undesirable rainy artifacts from a rainy image plays a crucial role for many high-level computer vision tasks and has drawn researchers’ attention from the past few years. The main drawbacks of Convolutional neural networks: have smaller receptive field, lack the model’s ability to capture long-range dependencies and complicated rainy artifacts, non-adaptive to input content and also computational complexity grows quadratically with input image size. These factors limit the deraining model performance improvement further. Recently, transformer has achieved better performance in both Natural language processing (NLP) and high-level computer vision (CV). We cannot adopt transformer directly to image deraining task as it has following limitations: a) although the transformer possesses powerful long-range computational capability, it lacks the ability to model local features b) to process input image, transformer uses fixed patch size, therefore pixels at the patch edges cannot use local features of surrounding pixels while removing heavy rain streaks. To address these issues, in single image deraining, we proposed a novel and efficient De-raining Transformer (DeTformer). In DeTformer, we designed a “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) to address the first issue and applied depthwise convolution to improve the modelling capability of local features and suppress unnecessary features and allow only useful information further. Also, the second issue was addressed, by introducing multi-resolution features in our network, and we applied progressive learning in the transformer and thus it allows the edge pixels to utilize local features effectively. Furthermore, to integrate the extracted multi-scale features and provide feature interaction across channel dimensions, we introduced a “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module. The proposed model experimented with various de-rained datasets and compared with various state-of-the-art models. The experimental results show that DeTformer network achieves superior performance compared to state-of-the-art networks on synthetic and real-world rain datasets.
Research Article
DeTformer: A Novel Efficient Transformer Framework for Image Deraining
https://doi.org/10.21203/rs.3.rs-2623687/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Computer vision (CV)
Deraining Transformer (DeTformer)
Gated-Depth-wise Convolution Feed-forward Network (GDWCFN)
Image Deraining
Multi-head Depth-wise Convolution Transposed Attention (MDWCTA)
Various image restoration tasks such as dehazing [1], inpainting [2] and image deraining [3] can improve image quality, and this helps to improve the detection accuracy of high-level CV tasks such as object classification and detection [4]. Therefore, image deraining has grabbed a lot of attention of researchers in this low-level CV area. Although many traditional algorithms were proposed to remove rain streaks from rainy images, yet it remains still complex and difficult, as there is no temporal information available in the captured images [5].
Consider the rainy image X, it can be expressed as a sum of rain layer R and background image B, and the physical model can be given as:
X = R + B (1)
Therefore, still image deraining is an ill-posed problem, since we know only X, and there are lots of solutions for both unknown B and R. Most of the existing networks have considered and focused to remove rain streaks via the optimization problem. Therefore, existing deraining networks falls in any of the two categories such as traditional model driven-based approaches and deep learning-based data-driven approaches.
Earlier researchers developed traditional model-driven based approaches such as sparse coding [11], decomposition [3], Gaussian Mixture models (GMM) [7] to remove rain streaks from rainy images. However, these traditional networks are very sensitive to image variations as they designed their networks using hand-crafted features. Due to the wide growth and great improvements in deep learning technology, currently researchers have moved to adopt new data-driven approaches like convolutional neural networks (CNN) [8] and transformer [20] for removing rain streaks. Compared to traditional model-driven approaches, deep learning-based data-driven approaches are more robust and achieve excellent results. Currently most of the data-driven based algorithms use CNN as their backbone to remove rain streaks. However, CNN have limited receptive fields and can capture only local spatial information and fail to capture broad contextual information.
To resolve this problem, some of the deraining networks introduced dilated convolution [6][30], or construct deeper networks [12–14] to enlarge CNN receptive fields. However, it still results in local information as the operation of convolution is just sliding a window and computing local weighted summation. If multiple convolutional layers were stacked, it just increases the network complexity, which leads to overfitting.
In recent years, transformer [15] was initially used for NLP task, and currently have been adopted in high-level computer vision tasks [4] and achieved impressive performance. CNN can only model local information, while transformer models the entire image and is adaptive to the input content. Greater success has been achieved in high-level CV tasks [16], so currently transformers have been adopted in low-level CV tasks such as dehazing, deraining networks [17–19].
U-shaped transformer [20] was proposed by Wang et al. by making refinements to Swin-Transformer [21]. A nested U-shaped transformer [31] was proposed just by increasing the number of transformer layers. However, we can adopt these transformers directly to single image deraining task as there are still many issues. (a) Transformer lacks the ability to model local features (b) to process input image transformer uses fixed patch size, therefore pixels at patch edges cannot use local features of surrounding pixels (c) Hierarchical encoder was incorporated in U-shaped transformer; and it couldn’t integrate multi-level features.
Therefore, we propose a deraining network based on transformer named as DeTformer, in order to explore and exploit the long-range contextual information during the complex single image deraining process. We introduce multi-scale features in image deraining in order to effectively utilize the efficacy of transformer fully. Therefore, it enables the transformer to use variable patch sizes and it also helps to improve the patch boundary defects. Second, a “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) module was introduced in transformer network to model local contextual information, as it plays a crucial role while removing rain streaks. A “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module was proposed for modelling long-range inter-pixel dependencies and also local information of multi-scale features extracted at different levels along channel-wise rather than spatial dimension. Wide experiments illustrate that our network not only generates clean images, but also helps in improving the efficiency of subsequent high-level CV tasks.
The contributions can be summarized as follows:
1. A novel efficient transformer based multi-scale structure was proposed for deraining single rainy images. Therefore, our network was able to model long-range inter-pixel contextual information in removing heavy and long rain streaks from rainy images.
2. We incorporated “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) in DeTformer and it utilizes local features to generate better rain-free images.
3. We designed “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module to integrate extracted multi-scale features effectively and also performs feature interaction along channels rather than spatial dimensions.
4. Experimental results demonstrate that DeTformer network outperforms SOTA networks on synthetic and real-rain world datasets.
In this section, a brief review on deraining methods is provided and such networks fall under either traditional model-driven or deep learning-based data-driven based approaches. Additionally, we provide previous works carried out by using multi-scale approaches and transformers.
Traditional model-driven prior based approaches
Traditional model-driven prior based approaches solve the image deraining process using prior knowledge. In [3], they decompose rain image into low and high frequency components, and adopted dictionary learning to remove high frequency rain components. Li et al. [7] proposed Gaussian mixture models for single image deraining. Discriminative sparse coding [11] used learning the dictionary of rain streaks and background layers during the image deraining. Low-rank model based traditional method was proposed by Chen et al. [23] to remove rain streaks and they assumed that rain streaks in a local patch have low-rank. Filter based sparsity and low-rank representation model was proposed by Zhang et al. [24] to remove rain streaks. In [36] a single image deraining model was proposed, which utilized proximal gradient descent technique and applied convolutional dictionary learning mechanism for rain representation. Although all the traditional based prior approaches have tried to achieve better results, they fail to remove rain streaks completely and cost time.
Deep learning-based data-driven approaches
Due to the wide success of deep learning in deraining [8-10], CNN based approaches has replaced the traditional model-driven prior based approaches for removing rain streaks. Therefore, researchers have designed many CNN-based network structure and proposed various loss functions to improve the performance of deraining networks.
Wang et al. [6] proposed a deep learning architecture to remove rain streaks from heavy rains. They created a model which contains two components for representation of rain streak accumulation and for representation of various shapes and directions of overlapping rain streaks. In [8] a density-aware deraining network was proposed, which identifies the rain streak densities and processes these streak densities effectively. “Generative adversarial network (GAN)” [10] was designed to remove rain streaks and generate derained images directly. X. Fu et al. [25] introduced deep CNN referred to as DerainNet to remove rain streaks. Wang H et al. [26] adopted image enhancement technique for deraining process and incorporated GAN to generate high quality rain patterns. To remove heavy rain streaks effectively, recursive networks [30][32][35] were adopted in deraining single images where the rain streaks were removed progressively and recursively.
Semi-supervised transfer learning technique [31] was adopted for single image deraining problem. This method uses semi-supervised and adds real rainy images without ground truth images into the network during training. Recursive operations were introduced on the top of a progressive ResNet in order to exploit the deep features across multiple stages and thus formed progressive recurrent network (PReNet) [32]. R. Yasarla et al. [33] proposed an over-and-under-complete CNN which pays special attention while learning local structures by employing receptive field of filters. In [34] a rain-to-rain auto encoder was proposed and rain embedding was introduced in the encoder to improve deraining performance. They also proposed layered LSTM for recursive recurrent deraining and feature refinement was performed at multiple scales by a fine-grained encoder. Fu X et al. [37] proposed rain streak removal via graph CNN to model long-range contextual information. Existing deraining networks embed low-quality features into the network directly, so Chen et al. [39] replaced low-quality features by high-quality features. They adopted closed-loop feedback control system to obtain latent high-quality features.
CNN based deraining networks have achieved unprecedented success when compared to traditional model-driven based networks. However, all these CNN-based deraining networks constructed by stacking multiple CNN layers and to model local information they uses their limited receptive field.
Vision Transformers
Initially, NLP adopted transformer and achieved great success. Transformer [16] was also designed for image classification and achieved better results than SOTA CNNs. To learn long-range inter-pixel dependencies between the sequences, attention [15] mechanism was applied and the images were split into patch sequences by transformer. As transformer possesses long-range modelling capability and adaptability to input content, they were adopted in various high-level CV tasks such as object classification, detection, tracking, segmentation and pose estimation. For image restoration, networks which adopted transformer are Restormer [19], U-former [20], Swin-IR [21], U2-former [22] and Transweather [27]. However, these networks perform poor on real-rain images which are affected by high-density rainfall. Therefore, in order to process high-resolution images, it requires huge computational complexity and also generate large number of parameters in transformer-based image deraining networks.
Multi‑scale pyramidal architecture
Using multi-scale learning, feature extraction would be improved to a certain extent since images of multiple scales can extract with different features. Light-weight pyramidal network [28] was developed using Gaussian-Laplacian image pyramid decomposition, and performs image deraining at each pyramid scale space. Jiang K et al. [29] constructed a pyramidal structure to improve the networks capability to encode rain streaks. Deep CNN based recurrent neural networks [30] were constructed to remove heavy rain streaks. They adopted dilated CNN to acquire large receptive field since contextual information plays a vital role during the image deraining process. To remove heavy rain, they incorporated squeeze-and-excitation network and decomposed rain removal into multiple stages and assigned them different alpha-values.
In [35], a combination of multi-scale feature fusion and progressive structure was introduced in their network to separate heavy rain streaks. To extract contextual information from the shallow layers they adopted U-net and at the last stage they incorporated image original resolution network to generate accurate derained images. A multi-stage architecture [38] was proposed which can progressively learn various image restoration functions for the degraded inputs. A supervised attention module was introduced to reweight local features by using per-pixel adaptive design. A “Deep feature interactive aggregation network” [40] was proposed to improve long-range pixel dependencies among the captured features and to build channel correlations among the features for image deraining.
Therefore, by introducing transformer, multi-scale information was added so it can exploit the advantages of the network global connectivity and also learn feature map representation in rain streaks.
Initially efficient transformer architecture was described and then followed by a brief description of individual components used in our network. To reduce computational complexity of a single-scale network [21] we made key changes to multi-scale hierarchical module and multi-head SA layer. The overall pipeline of DeTformer architecture is shown in Fig.1. A detailed description of core components of Transformer Block (TB) is as follows:
(a) “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module and
(b) “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN). At the end progressive training scheme and loss function details were provided.
First the degraded rainy image X € ℜ HxWx3 (HxW represents the spatial dimension and C represents the number of channels) is fed to a 3x3 convolution layer to obtain low-level features and then flatten the extracted features into “token”. Next these tokens i.e., shallow features pass via. Four-stage symmetrical encoder-decoder and are then transformed into deep features ℜ HxWx2C. Each stage of encoder-decoder contains a series of transformer blocks (TB) and to maintain efficiency of our network, we gradually increase the number of transformer blocks from top to bottom levels. Therefore, our encoder network only expands the channel capacity and hierarchically reduces spatial dimensions for the input image. A 4x4 convolution with stride-2 was performed during the down sampling operation so the number of channels was doubled and the feature map became half. The decoder network takes the low-resolution latent features ℜ H/8xW/8x8C and recovers progressively the high-resolution features. A 2x2 transposed convolution with stride-2 was performed during the up-sampling operation, so the number of channels reduces to half and the feature map becomes doubled.
We apply pixel-shuffle and pixel-unshuffled operations [41] for feature up-sampling and down-sampling. To make recovery process easy, we incorporated skip connections to concatenate encoder features with decoder features. After concatenation operation, we applied 1x1 convolution to make the number of channels become half at all stages, except at the top level. At stage-1, the low-level image features of encoder transformer block were aggregated with high-level features of decoder transformer block. Therefore, it helps to preserve the textural details and fine structures in output derained images. Now the deep features were enriched further in the refinement stage as it operates with high spatial resolution features. Finally, the refined feature map was fed to a 3x3 convolution layer to generate the residual feature map R € ℜ HxWx3 to which original rainy image X is added to reconstruct the derained image:
D = X + R.
Transformer Block (TB)
Each transformer block (TB) consists of dual layer normalization layers [42], one “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) and one “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) modules as shown in Fig. 2. Prior to MDWCTA and GDWCFN modules, Layer Normalization (LN) was applied and both modules perform element-wise addition using residual skip connections. It can be formulated as follows:
Feat1 = MDWCTA (LN(Feat0)) + Feat0 (2)
Feat2 = GDWCFN (LN(Feat1)) + Feat1 (3)
where LN refers to layer normalization, SA denotes self-attention, Feat0, Feat1 and Feat2 denote input feature map of TB, output feature map of MDWCTA and GDWCFN modules respectively.
Convolution Transposed Attention
The original transformer [16, 20] increases the computational complexity of the model as it globally calculates self-attention. We adopted “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) [21] in TB, in order to process high-resolution images while removing heavy rain streaks in the single image deraining task. Earlier works [43–44] adopted transformer and proved that they are deficient in processing local contextual information. Therefore, we replaced feed-forward network (FFN) [16, 21] in TBs with the proposed “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module. So, we compensate the transformers lack of capturing the local feature information with convolutional layers.
Multi-head Depth-wise Convolution Transposed Attention (MDWCTA)
The transformer computational burden increase mainly comes from the self-attention (SA) layer. In the original transformer [16, 20], the memory and time complexity of key-query dot-product interaction increases quadratically with the spatial resolution of input i.e. O(W2H2). Therefore, it is not quite feasible to apply SA on image deraining tasks as it often involves high resolution images.
We proposed MDWCTA module which has a linear complex structure to resolve this issue as shown in Fig. 2(a)In this module, they apply SA along the channel dimensions instead of spatial dimensions i.e., cross-covariance is computed across the channels and generates attention map encoding the global context by default. One key change we made in this module, a 3x3 depth-wise convolution was introduced to highlight the local contextual information prior to feature covariance computing in order to produce global attention map.
To reduce computational complexity burden in the proposed network, and to perform self-attention, a “non-overlap window- based” technique was applied. On the input feature map F € (H x W x C), the layer normalized tensor generates (HW/M2) x C local feature maps, as M x M local-window slice was applied. Here (HW/M2) is the total divided windows. The obtained local features map was enriched as 1x1 convolution was applied to aggregate pixel-wise cross-channel contextual information. Then 3x3 depthwise convolutions were applied to encode the channel-wise spatial contextual information, which yields normalized feature map, and the matrices for query (Q), key (K) and value (V) are given by:
Q = FXQPQ, K = FXKPK, V = FXVPV (4)
Where P and X perform 1×1 point-wise convolution and 3×3 depth-wise convolution. In the proposed network we don’t use bias in convolutional layers. A transposed attention map A € ℜ CxC was generated instead of larger regular attention map A € ℜ HW x HW by reshaping the query and key pair projections. The overall process of MDWCTA is formulated as follows:
Feat1 = P. (Attention (Q^, K^, V^)) + Feat0 (5)
Attention (Q^, K^, V^) = V^. Softmax (K^. Q^/α) (6)
Where Feat0 , Feat1 are input and output feature maps, α is a learning scalable parameter which is used to control the magnitude of K^. Q^ before applying softmax. In our module the number of channels was divided into heads and it learns separate attention maps paralelly which is similar to conventional multi-head SA [16].
Gated-Depth-wise Convolution Feed-forward Network (GDWCFN)
Figure 2. (b) shows the GDWCFN architecture. A regular FFN [16] operates on each pixel separately and identically while transforming the image features. They used two 1x1 convolutions initially, one to expand feature channels and other to reduce the channels to get back the original image size. Therefore, we applied a non-linearity function in the hidden layers. To improve representation learning we made two modifications to a regular FFN. One is gated mechanism which was incorporated and the other one adopted depthwise convolutions.
To perform element-wise product of two parallel paths of linear transformation layers, a gating mechanism was formulated, and one of which was activated with non-linear GeLU [45]. As in MDWCTA, we adopted 3x3 depthwise convolutions in GDWCFN to encode information from the spatially neighbouring pixel positions, as it is useful to learn local image structures.
For an input tensor Y € ℜ H x W x C, GDWFCN was formulated as follows:
Y^ = P Gating(Y) + Y (7)
Gating(Y) = µ(Wd1Wp1(LN(Y))) ȇ Wd2Wp2 (LN(Y)) (8)
where µ represents non-linear GeLU function, ȇ denotes element-wise multiplication and LN is layer normalization. The proposed network GDWCFN, controls the information flow through multi-hierarchical levels and it allows each stage to put focus only on the fine details inverted to other stages. Therefore, this module plays a more vital role compared to MDWCTA module as its focus is to enrich the features with contextual information.
Progressive Learning
Many existing CNN based deraining networks usually train the networks by using image patches of fixed-size. However, the original transformer model [17] trained on small cropped patches could not achieve optimal performance during image restoration. Therefore, we implemented progressive learning strategy where DeTformer network was trained initially with small image patches in the early epochs and gradually, patch sizes were increased in later epochs. As we adopted mixed-size image patches training strategy, we were able to achieve better results during testing even for high-resolution images. Therefore, our network was able to preserve the fine image structures and texture while removing rain streaks as our network was trained using a curriculum learning fashion. We reduced the batch size as the patch size increased while training on large patches since it consumes longer time than usual. We needed to maintain similar time as fixed patch training.
Loss function
In order to train deep draining networks, the widely adopted loss functions are Mean absolute error (L1) loss, Mean square error (L2) loss, negative SSIM loss, Charbonnier loss, attention loss, edge loss, adversarial loss and perceptual loss. We adopted Charbonnier loss in the proposed network as it makes the model converge faster and can tolerate small errors. The total loss function is expressed as:
Where ∆ is Gaussian operator which can control the relative importance of the loss terms in Eq. (9), λ (hyper parameter) was set to 0.05 and ε constant was set to 10− 3.
Here we provide details of our experimental setup, datasets and performance metrics. We evaluated the performance and showed the effectiveness of DeTformer network on benchmark synthetic and real-rain datasets.
a) Experimental setup
Our proposed network was implemented on PyTorch 1.7 deep learning framework. AdamW optimizer solution was applied during the network training for a total of 105 iterations. Fixed learning strategy was used with 3 × 10−4 learning rate. Batch size was set to 8, and adapted variable patch sizes 128x128, 160x160 and 192x192 respectively. To make proposed network more robust various augmentation techniques were applied such as horizontal flip and vertical flip during the network training. In all TBs, window size was fixed to 8. All the experiments were carried on Google Colab pro which has Tesla V100 16G GPU.
b) Datasets
The effectiveness of the proposed network was evaluated on synthetic paired rain datasets and real-rain dataset [28], which includes Rain100L [6], Rain100H [6], Rain800 [10], Rain1200 [8], Rain12 [7] and Rain14000 [9] and renamed Testset as Rain100L, Rain100H, Test100, Test1200 and Test2800. Table.1 shows a brief summary of datasets used in this work.
Table.1. Summary of used datasets.
|
Datasets |
Train Imgs. |
Test Imgs. |
Testset Renamed |
|
Rain800 [10] |
700 |
100 |
Test100 |
|
Rain14000 [9] |
11200 |
2800 |
Test2800 |
|
Rain1800 [6] |
1800 |
0 |
NC |
|
Rain100L [6] |
0 |
100 |
Rain100L |
|
Rain100H [6] |
0 |
100 |
Rain100H |
|
Rain1200 [8] |
0 |
1200 |
Test1200 |
|
Rain12 [7] |
12 |
0 |
NC |
|
Total |
13712 |
4300 |
|
c) Evaluation metrics
To show the effectiveness and performance of the proposed network, we evaluated the derained image quality using two evaluation metrics. “Peak-signal-to-noise-ratio” (PSNR) and “Structural similarity index measurement” (SSIM) were calculated on the restored images. Generally, the larger their values are, the better the deraining effect is.
4.1. Comparison with the state‑of‑the‑art networks
We compared the performance of DeTformer comprehensively with several state-of-the-art (SOTA) deraining networks such as JORDER [6], DID-MDN [8], RESCAN [30], SSTL [17], PReNet [32], DerainNet [25], UMRL [26], MSPNet [29], SAPNet [46], SEMI [31], OUCD [33], ECNet [34], PMSDNet [35], RCDNet [36], DualGCN [37], MPRNet [38], RLNet [39] and DFIANet [40].
The visual quantitative results of DeTformer network on synthetic rain datasets are shown in Table.2. It is clear from the table that our proposed network achieves superior performance over all the state-of-the-art (SOTA) networks on all synthetic datasets. In particular, on Rain100L and Rain100H datasets, DeTformer network obtains 38.99 and 21.45 dB PSNR which is +3.79 and +1.97 dB PSNR higher compared to DFIANet [40] and which clearly shows that our network removes heavy and complex rain streaks more effectively. Table.2, shows that DeTformer network achieves the highest PSNR and SSIM metric values on Rain100L, Rain100H, Test100, Test1200 and Test2800 synthetic datasets. These is due to the fact that our network uses the benefits of transformer as well models the long-range contextual information better.
The visual qualitative results of the proposed network on synthetic rain datasets are shown in Fig.3, Fig.4 and Fig.5 respectively. Although the networks (PReNet, ECNet and DFIANet) remove the heavy rain streaks but yet “visible artifacts” and “blurred details” were observed in the derained outputs as shown in Fig.3.
Table.2 Quantitative results of the proposed network on synthetic datasets and made comparison with the SOTA networks.
|
Datasets |
Year |
Rain100L [28] |
Rain100H [28] |
Test100 [27] |
Test2800 [13] |
Test1200 [15] |
Avg. PSNR/ |
|
Metrics |
PSNR/SSIM |
PSNR/SSIM |
PSNR/SSIM |
PSNR/SSIM |
PSNR/SSIM |
Avg. SSIM |
|
|
Networks |
|
|
|
|
|
|
|
|
JORDER [6] |
2017 |
31.27/0.92 |
27.75/0.84 |
24.72/0.85 |
31.6/0.91 |
31.27/0.90 |
29.32/0.88 |
|
DIDMDN [8] |
2018 |
25.23/0.74 |
17.35/0.52 |
22.56/0.82 |
28.13/0.86 |
29.65/0.90 |
24.58/0.77 |
|
RESCAN [30] |
2018 |
29.80/0.88 |
26.36/0.78 |
25.01/0.83 |
31.29/0.90 |
30.51/0.88 |
28.59/0.85 |
|
SSTL [17] |
2019 |
25.03/0.84 |
16.56/0.48 |
22.35/0.78 |
24.43/0.78 |
26.05/0.82 |
22.88/0.74 |
|
PReNet [32] |
2019 |
32.44/0.95 |
26.77/0.85 |
24.81/0.85 |
31.75/0.91 |
31.36/0.91 |
29.42/0.89 |
|
DerainNet [25] |
2017 |
27.03/0.88 |
14.92/0.59 |
22.77/0.81 |
24.31/0.86 |
23.38/0.83 |
22.48/0.79 |
|
UMRL [26] |
2019 |
29.18/0.92 |
26.01/0.83 |
24.41/0.83 |
29.97/0.90 |
30.55/0.91 |
28.02/0.88 |
|
MSPNet [29] |
2020 |
32.44/0.95 |
28.66/0.86 |
27.50/0.87 |
32.82/0.93 |
32.39/0.91 |
30.76/0.74 |
|
SEMI [31] |
2019 |
22.25/0.84 |
18.08/0.57 |
20.72/0.68 |
24.38/0.73 |
23.91/0.71 |
21.87/0.70 |
|
OUCD [33] |
2021 |
29.84/0.90 |
24.38/0.73 |
23.58/0.80 |
28.72/0.89 |
26.09/0.82 |
26.52/0.83 |
|
ECNet [34] |
2022 |
33.42/0.95 |
27.91/0.86 |
27.55/0.88 |
32.42/0.93 |
30.05/0.90 |
30.27/0.90 |
|
PMSDNet [35] |
2022 |
36.41/0.97 |
30.38/0.89 |
30.32/0.90 |
33.62/0.93 |
32.96/0.92 |
32.74/0.92 |
|
RCDNet [36] |
2020 |
38.60/0.98 |
28.83/0.88 |
24.59/0.82 |
- |
29.81/0.86 |
30.45/0.88 |
|
DualGCN [37] |
2021 |
38.05/0.99 |
29.06/0.91 |
28.28/0.89 |
- |
32.98/0.93 |
25.67/0.93 |
|
MPRNet [38] |
2021 |
36.69/0.97 |
27.65/0.87 |
27.86/0.85 |
- |
31.73/0.91 |
30.98/0.90 |
|
RLNet [39] |
2021 |
37.38/0.98 |
28.87/0.90 |
27.95/0.87 |
- |
32.62/0.91 |
31.70/0.91 |
|
SAPNet [46] |
2022 |
34.77/0.97 |
29.46/0.89 |
29.13/0.88 |
32.18/0.93 |
32.46/0.91 |
31.60/0.91 |
|
DFIANet [40] |
2022 |
35.20/0.95 |
29.48/0.87 |
28.90/0.88 |
33.12/0.93 |
32.92/0.92 |
31.92/0.91 |
|
Proposed |
|
38.99/0.97 |
31.45/0.90 |
32.07/0.90 |
34.17/0.94 |
33.18/0.92 |
33.97/0.93 |
From the observation of derained images in Fig.3, this situation occurs in clouds, sky and roof and appears in JORDER [6], RESCAN [30], SEMI [31] and DFIANet [40] networks. As the colour of background is similar to rain streaks, some networks perform excessive deraining and remove the fine details of similar colour as in the second row of Fig.3.When the test images contain denser objects, it is difficult to remove rain streaks completely and recover finer details simultaneously, was clear from the telephone booth and black fence in the third and fourth rows in SEMI [31], PReNet [32], ECNet [34], JORDER [6] and DFIANet [40]. OUCD [33] network combines global information in their network and pays more attention only on local features and the network fails to remove heavy rain streaks completely. Therefore, compared to all these SOTA networks our proposed network can avoid all these problems and restore the derained images which is highly similar to target images.
Fig.4. shows that our network exhibits impressive recovery deraining results while removing diverse rain and also heavy rain. From the observed images in the 10th column, our network was able to restore clear image details and appropriate contrast and which are similar to ground truth images. Some more sample deraining results of the proposed network on Rain100H synthetic dataset along with their Mean square error (MSE), PSNR and SSIM are shown in Fig.5.
To show the robustness and efficiency of the proposed network, we also made a comparison with SOTA networks on real-rain dataset [28]. Fig.6 shows the derained results on real-rain dataset of the proposed network and a comparative analysis with DualGCN [37], MPRNet [38], RLNet [39] and DFIANet [40] networks. However, many of these methods produce artifacts during the image deraining process, which are not as clear as that of the images restored by our network. Our network removes rain streaks which are more unevenly distributed, and also achieves impressive performance while removing heavy rain streaks and outputs clear and detailed content results. Inspite of complex rain scenes present in nature, our network generates excellent results while removing rain streaks under realistic conditions.
We also provided a number of parameters required and floating-point operations (FLOPS) performed on a specific Rain100H dataset and made comparison with the SOTA networks in Table.3. It is observed that number of parameters in the proposed network reduces as general convolution was replaced by transformer.
Table.3. Number of parameters and FLOPS generated by SOTA networks.
|
Network |
FLOPS (G) |
Parameters (M) |
|
MPRNet [38] |
175.80 |
28.46 |
|
SwinIR [25] |
238.00 |
36.94 |
|
Uformer [20] |
174.70 |
32.58 |
|
HiNet [47] |
293.79 |
26.59 |
|
Proposed |
87.70 |
25.31 |
4.2. Ablation studies
A series of ablation studies were conducted to show the impact of various factors on DeTformer network and we evaluated the ability of the proposed network during the deraining process. All ablation studies use Rain100H during network training and testing.
4.2.1. Effect of basic composition
Table.4 shows the ablation study results of the importance of each component separately. Therefore, our network achieves higher quality performance. As seen from the table when FFN was replaced with GDWCFN module, PSNR dropped by 0.88 dB. So, this proves the effectiveness of GDWCFN in enhancing and preserving the local feature information and alleviates the drawback of original transformer in extracting local feature information. If MDWCTA module is removed, PSNR drops by 1.31 dB and this proved that the networks performance would be improved by multi-scale feature fusion. PSNR was drastically reduced by 1.52 dB, when all the up sampling and down sampling layers were removed and this shows the effectiveness of the designed U-shaped transformer structure. We also provided a number of parameters required and floating-point operations (FLOPS) performed when a specific component was employed in the proposed network.
Table.4. Effects of basic composition in the proposed network.
|
Component |
PSNR |
FLOPS (B) |
Parameters (M) |
|
Remove MDWCTA |
30.14 |
83.7 |
24.86 |
|
Remove GDWCFN with FFN |
29.57 |
85.3 |
25.02 |
|
Replace TB with CNN |
27.86 |
81.3 |
23.02 |
|
Eliminate up and down sampling layers |
28.93 |
83.4 |
24.53 |
|
Proposed structure |
31.45 |
87.7 |
25.31 |
We also performed experiments on the number of scales to be employed in the encoder-decoder network structure for removing rain streaks effectively during the deraining process.
4.2.2. Effect of number of scales
Table.5 shows the impact of the number of scales to be employed in the proposed network and to show the effectiveness of multi-scale structure. From these observations, it is clear that when S=1, PSNR drops by 0.24 dB, since multi-resolution features can assist the DeTformer network better to remove heavy and complex rain streaks effectively. When S=4 we were able to achieve both higher PSNR and SSIM metric values.
Table.5. Impact of number of scales in the proposed network.
|
No. of scales |
PSNR |
SSIM |
|
1 |
30.21 |
0.87 |
|
2 |
31.13 |
0.88 |
|
3 |
31.26 |
0.89 |
|
4 |
31.45 |
0.9 |
4.2.3. Effect of λ hyper parameter
From equation (9) the total weighted loss function depends on λ hyper parameter and was set to 0.05. In order to obtain better network performance, we performed an ablation study to fix λ parameter. Table.6 shows the influence of λ value on PSNR and SSIM values. So from these observations, we fixed λ value as 0.05 in weight loss function as it achieves higher PSNR and SSIM.
Table.6. Impact of λ parameter on total loss function.
|
λ |
PSNR |
SSIM |
|
0 |
31.16 |
0.89 |
|
0.05 |
31.45 |
0.9 |
|
1 |
31.32 |
0.9 |
|
2 |
30.97 |
0.88 |
4.2.4. Effect of number of transformer blocks in encoder-decoder network
To decide the number of transformer blocks (N) to be employed in the encoder-decoder network, we performed an ablation study. Table.7 shows the impact of N on the proposed network on complexity and computational burden. In order to balance both complex structure and computational complexity i.e., deraining performance and efficacy we adopt N = 2 in our network.
Table.7. Impact of N in the proposed network.
|
N |
PSNR |
SSIM |
|
1 |
31.26 |
0.89 |
|
2 |
31.45 |
0.90 |
|
3 |
32.94 |
0.91 |
|
4 |
33.27 |
0.91 |
4.2.5. Effect of different loss functions in the proposed network
An ablation study was conducted to show the effectiveness of Charbonnier loss, and make a comparison with two other popular loss functions L1 and L2. Table.8 shows the effectiveness of Charbonnier loss, so we adopted this loss function to reconstruct the derained image.
Table.8. Effect of loss function for improving deraining performance.
|
Loss function |
PSNR |
SSIM |
|
L1 |
31.36 |
0.89 |
|
L2 |
31.39 |
0.89 |
|
Charbonnier |
31.45 |
0.90 |
4.2.6. Impact of progressive learning
The impact of progressive learning adopted in our network ablation study is shown in Table.9. We achieved better results using progressive learning than fixed patch training while still balancing similar training time.
Table.9. Impact of progressive learning on the proposed network
|
Patch size |
PSNR |
Train time (Hours) |
|
Progressive (1282 to 1922) |
31.45 |
23.4 |
|
Fixed (1282) |
31.33 |
22.3 |
4.3. Limitation
Although our DeTformer deraining network has achieved superior performance over SOTA networks, it has certain limitations. During the testing stage we fed our network with a rain-drop image and it shows inconsistent behaviour and was unable to remove rain drops as shown in Fig.7. This is because we didn’t train the proposed network with rain-drop images.
We present a transformer based deraining network referred as DeTformer. To process more complex and realistic rainy images and restore fine details, we proposed an efficient DeTformer network and also made comparative analysis with SOTA deraining networks. The superior performance of the proposed network was achieved by a series of improvements. In this work, transformer structure was adopted in deraining single images. We designed “Gated-Depth-wise Convolution Feed-forward Network” (GDWCFN) and applied depth-wise convolution which can improve the capability of modeling local features and suppresses less informative features. We incorporated multi-resolution features in the transformer, where the proposed network can use patches of different scales, and thus it enables the edge pixels to utilize local features. Furthermore, we designed “Multi-head Depth-wise Convolution Transposed Attention” (MDWCTA) module which can effectively integrate the multi-scale extracted features and also perform feature interaction across channel dimensions. Extensive experiments of the proposed network demonstrate that it achieves superior performance on both synthetic-paired and real-rain datasets.
Conflict of interest: The authors declare that they have no conflict of interest.
Data availability statements: All data generated or analysed during this study are included in this published articles Rain100L [6], Rain100H[6 ], Rain1800[6], Rain12 [7], Rain1200[8], Rain14000 [9], Rain800 [10].
- Wang, T., Zhao, L., Huang, P., Zhang, X., & Xu, J. (2021). Haze concentration adaptive network for image dehazing. Neurocomputing, 439, 75-85.
- Ahmed, M. W., & Abdulla, A. A. (2020). Quality improvement for exemplar-based image inpainting using a modified searching mechanism. UHD Journal of Science and Technology, 4(1), 1-8.
- Kang, L. W., Lin, C. W., & Fu, Y. H. (2011). Automatic single-image-based rain streaks removal via image decomposition. IEEE transactions on image processing, 21(4), 1742-1755.
- Murthy, C. B., Hashmi, M. F., & Keskar, A. G. (2022). EfficientLiteDet: a real-time pedestrian and vehicle detection algorithm. Machine Vision and Applications, 33(3), 47.
- Yang, W., Tan, R. T., Wang, S., Fang, Y., & Liu, J. (2020). Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on pattern analysis and machine intelligence, 43(11), 4059-4077.
- Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017). Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1357-1366).
- Li, Y., Tan, R. T., Guo, X., Lu, J., & Brown, M. S. (2016). Rain streak removal using layer priors. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2736-2744).
- Zhang, H., & Patel, V. M. (2018). Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 695-704).
- Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., & Paisley, J. (2017). Removing rain from single images via a deep detail network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3855-3863).
- Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(11), 3943-3956.
- Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE international conference on computer vision (pp. 3397-3405).
- Luo, Y., Xu, Y., & Ji, H. (2015). Removing rain from a single image via discriminative sparse coding. In Proceedings of the IEEE international conference on computer vision (pp. 3397-3405).
- Deng, S., Wei, M., Wang, J., Feng, Y., Liang, L., Xie, H., ... & Wang, M. (2020). Detail-recovery image deraining via context aggregation networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14560-14569).
- Yang, H., Zhou, D., Li, M., & Zhao, Q. (2022). A two-stage network with wavelet transformation for single-image deraining. The Visual Computer, 1-17.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- Tan, F., Kong, Y., Fan, Y., Liu, F., Zhou, D., Chen, L., ... & Qian, Y. (2021). SDNet: mutil-branch for single image deraining using swin. arXiv preprint arXiv:2105.15077.
- Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., ... & Gao, W. (2021). Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12299-12310).
- Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., & Yang, M. H. (2022). Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5728-5739).
- Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., & Li, H. (2022). Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17683-17693).
- Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 10012-10022).
- Ji, H., Feng, X., Pei, W., Li, J., & Lu, G. (2021). U2-former: A nested u-shaped transformer for image restoration. arXiv preprint arXiv:2112.02279.
- Chen, Y. L., & Hsu, C. T. (2013). A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In Proceedings of the IEEE international conference on computer vision (pp. 1968-1975).
- Zhang, H., & Patel, V. M. (2017, March). Convolutional sparse and low-rank coding-based rain streak removal. In 2017 IEEE Winter conference on applications of computer vision (WACV) (pp. 1259-1267). IEEE.
- Fu, X., Huang, J., Ding, X., Liao, Y., & Paisley, J. (2017). Clearing the skies: a deep network architecture for single-image rain removal. 26 (6): 2944–2956. DOI: https://doi. org/10.1109/TIP.
- Yasarla, R., & Patel, V. M. (2019). Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8405-8414).
- Valanarasu, J. M. J., Yasarla, R., & Patel, V. M. (2022). Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2353-2363).
- Fu, X., Liang, B., Huang, Y., Ding, X., & Paisley, J. (2019). Lightweight pyramid networks for image deraining. IEEE transactions on neural networks and learning systems, 31(6), 1794-1807.
- Jiang, K., Wang, Z., Yi, P., Chen, C., Huang, B., Luo, Y., ... & Jiang, J. (2020). Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8346-8355).
- Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In Proceedings of the European conference on computer vision (ECCV) (pp. 254-269).
- Wei, W., Meng, D., Zhao, Q., Xu, Z., & Wu, Y. (2019). Semi-supervised transfer learning for image rain removal. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3877-3886).
- Ren, D., Zuo, W., Hu, Q., Zhu, P., & Meng, D. (2019). Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3937-3946).
- Yasarla, R., Valanarasu, J. M. J., & Patel, V. M. (2020). Exploring overcomplete representations for single image deraining using cnns. IEEE Journal of Selected Topics in Signal Processing, 15(2), 229-239.
- Li, Y., Monno, Y., & Okutomi, M. (2022). Single image deraining network with rain embedding consistency and layered LSTM. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4060-4069).
- Ragini, T., & Prakash, K. (2022, December). Progressive Multi-scale Deraining Network. In 2022 IEEE International Symposium on Smart Electronic Systems (iSES) (pp. 231-235). IEEE.
- Wang, H., Xie, Q., Zhao, Q., & Meng, D. (2020). A model-driven deep neural network for single image rain removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3103-3112).
- Fu, X., Qi, Q., Zha, Z. J., Zhu, Y., & Ding, X. (2021, May). Rain streak removal via dual graph convolutional network. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 2, pp. 1352-1360).
- Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M. H., & Shao, L. (2021). Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 14821-14831).
- Chen, C., & Li, H. (2021). Robust representation learning with feedback for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7742-7751).
- Cao, S., Liu, L., Zhao, L., Xu, Y., Xu, J., & Zhang, X. (2022). Deep Feature Interactive Aggregation Network for Single Image Deraining. IEEE Access, 10, 103872-103879.
- Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., ... & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1874-1883).
- Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.
- Li, Y., Zhang, K., Cao, J., Timofte, R., & Van Gool, L. (2021). Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707.
- Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021). Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 22-31).
- Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
- Zheng, S., Lu, C., Wu, Y., & Gupta, G. (2022). SAPNet: Segmentation-aware progressive network for perceptual contrastive deraining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 52-62).
- Chen, L., Lu, X., Zhang, J., Chu, X., & Chen, C. (2021). Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 182-192).
No competing interests reported.