Deep learning-based diagnosis systems are useful to identify abnormalities on medical images with the greatly increased workload of doctors. Specifically, the rate of new cases and deaths from malignancies is rising for liver diseases. Early detection of liver lesions plays an extremely important role in effective treatment and gives a higher chance of survival for patients. Therefore, automatic detection and classification of common liver lesions are essential for doctors. In fact, radiologists mainly rely on Hounsfield Units to locate liver lesions but the previous studies often pay little attention to this factor. In this paper, we propose an improved method for automatic classification of common liver lesions based on deep learning technique and the variation of Hounsfield Unit densities on CT images with and without contrast. Hounsfield Unit is used to locate liver lesions accurately and support data labeling for classification. We construct a multi-phase classification model developed on the deep neural networks of Faster R-CNN, R-FCN, SSD, and Mask R-CNN with transfer learning approach. The experiments are conducted on six scenarios with multi-phase CT images of common liver lesions. Experimental results show that the proposed method improves the detection and classification of liver lesions compared with recent methods because its accuracy achieves up to 97.4%. It is useful for practical systems to assist doctors in the diagnosis and treatment of liver lesions.