The oxygen pump is a complex component of liquid rocket engine, and is also one of the main vibration sources of engine. In this paper, the unsteady fluid exciting load was studied considered the coupling of gap flow and main flow. The results were verified through test of liquid rocket engine. Modal characteristics of oxygen pump case-volute with different liquid medium filled was tested. The Rayleigh damping parameter was derived by considering the frequency characteristics of fluid excited. Thus, a complete transient dynamic model was established. The transient dynamic response of oxygen pump case-volute under unsteady fluid excited was studied. At the same time, the time-domain and frequency-domain characteristics of vibration response were revealed. The analysis method was verified by comparing with the statistical distribution of test results. The vibration response of oxygen pump case-volute mainly shows frequency doubling of rotational speed, among which the 12X and 18X frequencies are prominent.