Internet-of-Vehicles (IoV) plays an important part of Intelligent Transportation Systems, and is widely regarded as one of the most strategic applications in smart cities development. Next generation wireless network is especially crucial for meeting the connectivity and bandwidth demands of IoVs. Smart spectrum resource management has received much attention of the research community as it is believed to be a promising approach for solving the spectrum resource challenge of IoV and Intelligent Transportation Systems. In this article, we propose a smart spectrum optimization technique based on a deep learning method for user mobility prediction. For this purpose, based on the Exploration and Preferential Return (EPR) model which can be used to investigate the movement trend and aggregation behavior of the target, we adopt the D-Exploration and Preferential Return (D-EPR) model as a deep learning technique to train a Long-Short Term Memory (LSTM) recurrent neural network (RNN) in order to predict the future locations of IoV nodes. With predicted user’s mobility, a graph theoretic algorithm is then applied to achieve spectrum reuse and optimization. Besides, our proposed deep-learningbased user mobility prediction is able to identify the user position. This paper then compares the performances of mobility prediction by traditional method and our proposal. The outcomes of spectrum efficiency and network capacity are also provided to show the effectiveness of the proposed solution.