Accurate and reliable monthly runoff forecasting plays an important role in making full use of water resources. In recent years, long short-term memory neural networks (LSTM), as a deep learning technology, has been successfully applied in forecasting monthly runoff. However, the hyperparameters of LSTM is predetermined, which has a significant influence on model performance. In this study, given decomposition of monthly runoff series may provide more accurate predication revealed by many previous studies, a hybrid model, namely VMD-GWO-LSTM, has been proposed for monthly runoff forecasting. The proposed hybrid model is comprised of two main components, namely variational mode decomposition (VMD) coupled with grey wolf optimizer (GWO) based LSTM. First, VMD is utilized to decompose raw monthly runoff series into several subsequences. Second, GWO is implemented to optimize the hyperparameters of LSTM for each subsequence on condition that the inputs are determined. Finally, the total output of all subsequences is aggregated as final forecast result. Four quantitative indexes are employed to evaluate the model performance. The proposed model is demonstrated using monthly runoff series data derived from two reservoirs in China's Pearl River system. To identify the feasibility and superiority of the proposed model, back propagation neural networks (BPNN), support vector machine (SVM), LSTM, EMD-LSTM, VMD-LSTM and GWO-LSTM are also utilized for comparison. The results indicate that the proposed hybrid model can yield best forecast accuracy among these models, making it a promising new method for monthly runoff forecasting.