In this work, a multi-scale residual dense network (MSRDN) with dilatedconvolution is proposed to efficiently eliminate noise in ECG signals.Based on dilated convolutions with different sampling rate, a dual-branchresidual dense block (DBRDB) is designed to extract multi-scale localfeatures, and dual-way feature fusion increases the variation of informa-tion flow input to the subsequent blocks whilst requiring fewer parame-ters. The hierarchical feature-maps learned by all the DBRDBs can beadaptively fused and further combined with the shallow features to con-stitute the global features. And the residual learning is also introducedinto MSRDN to achieve cross-layer information interaction and acceler-ate network training. We evaluate our model on the MIT-BIH arrhythmiadatabase and the MIT-BIH noise stress test database. The results of theexperiments show that our method outperforms the existing traditionaland deep learning methods in terms of the performance metrics of SNR,RMSE and PRD. And denoised waveforms are closer to the originalclean signals while preserving important details of the ECG signals.