Virtual cleaning of art is a key process that conservators apply to see the likely appearance of the work of art they have aimed to clean, before the process of cleaning. It is also of public interest, allowing people to see their favorite work of art without the yellow tint of the varnish which has covered the work. There have been many different approaches to virtually clean artwork, all of which need to physically clean the work in at least a few spots, an impediment in some cases. Another issue regarding the methods devised until now is that some of them need samples of pure white or black paint on the painting which might be, again, an obstacle as there are certainly works without any pure black and white paint. To overcome these shortcomings, a Convolutional Neural Network is trained on representative RGB images; the images are first artificially yellowed using a physics based model and the network is trained to go from the yellowed images to the original colored images. The results show improvement over a proposed method using a physical model to remove the impacts of age. The CNN is then applied to images of the Mona Lisa and The Virgin and Child with Saint Anne, both painted by Leonardo da Vinci, and works for which we have images from both before and after physical cleaning. Results show both a qualitative and quantitative improvement in the color quality of the resulting image. The novelty of the work proposed herein lies in two premises. First is the accuracy of the method, which is demonstrated through comparing the method with the only physical approach derived until now. Second is the generalizability of the method which is shown through blindly applying the method to two famous works of art for which no information but an RGB image of the uncleaned artwork is known.