The parts with integrated design technology are widely used in aerospace field because of their advantages such as high strength and high reliability compared with riveted structures, which often present multiple cavities and thin walls, and milling occupies a large amount of machining time, while it is very easy to cause rapid degradation of tool performance because their materials are mostly made of hard-to-machine metals. To address the failure of traditional methods in monitoring the tool trajectory of complex cavity milling process, this paper proposes a tool wear prediction method based on short-time Fourier transform and improved depth residual network. Firstly, the short-time Fourier transform is used to convert the signal into a time-frequency map. Then, to solve the problem that the depth residual network describes the machining state from a single perspective, the original model is improved by adding a feature fusion layer. Finally, the signal and the time-frequency map are simultaneously input into the improved deep residual network, and the tool wear sensitive features filtered by Pearson correlation coefficient are extracted from the time domain and frequency domain of the signal, and the deep features in the time-frequency domain are extracted from the residual block structure of the time-frequency map, and the tool wear sensitive features and the deep features in the time-frequency domain are fused in the feature fusion layer to complete the model training. The experimental results show that the average prediction deviation of tool wear by the regression model established in this paper is 0.76%, which is lower than that of the original deep residual network, shallow convolutional neural network and artificial feature-based machine learning model.