Extracting traffic information from images plays an important role in Internet of Vehicle (IoV). However, due to the high-speed movement and bumpiness of the vehicle, motion blur will occur in image acquisition. In addition, in rainy days, because the rain is attached to the lens, the target will be blocked by rain, and the image will be distorted. These problems have caused great obstacles for extracting key information from transportation images, which will affect the real-time judgment of vehicle control system on road conditions, and further cause decision-making errors of the system and even cause traffic accidents. In this paper, we propose a motion blurred restoration and rain removal algorithm for IoV based on Generative Adversarial Network (GAN) and transfer learning. Dynamic scene deblurring and image de-raining are both among the challenging classical tasks in low-level vision tasks. For both tasks, firstly, instead of using ReLU in a conventional residual block, we designed a residual block containing three 256-channel convolutional layers, and we used the Leakly-ReLU activation function. Secondly, we used generative adversarial networks for the image deblurring task with our Resblock, as well as the image de-raining task. Thirdly, experimental results on the synthetic blur dataset GOPRO and the real blur dataset RealBlur confirm the effectiveness of our model for image deblurring. Finally, we can use the pre-trained model for the transfer learning-based image de-raining task and show good results on several datasets.