Existing low-light image enhancement techniques are mostly not only difficult to deal with both visual quality and computational efficiency but also are prone to detail loss and low contrast. in unknown complex scenarios. To solve these problems, we proposed an efficient low-light enhancement network for Removing the Darkness (denoted as RtD-Net). The network decomposes the image into two parts, the illumination branch is used to deal with light regulation and the reflection branch is used to deal with degradation removal. In this paper, we introduce the structure of RtD-Net, loss function, and training method of the model. First, the residual layer module is added to the network structure to improve the feature learning performance and the detail processing ability of the network; then a composite function including image reconstitution loss function, SSIM (Structural Similarity) loss function and TV loss function is used to optimize the contrast, brightness, color and saturation of the low-light image; Finally, segmental training is used to further improve the quality of the generated enhanced images. In the experimental comparison, the model demonstrates more advanced performance in both qualitative and quantitative comparisons, and is in an advantageous position when comparing with other low-light image enhancement models for image quality metrics. It can be proved through experiments that the proposed method has reached the state-of-the art in terms of processing speed and effect.