Due to long-term mining, a series of high and steep rock slopes have been formed in the open-pit mine. For high rock slopes, rainfall infiltration is the main cause of landslide. Therefore, the stability analysis of high rock slope under rainfall has become a key issue in the open-pit mine engineering. In this work, aiming at the high stress condition of high rock slope, the instantaneous internal friction angle and instantaneous cohesion of rock mass under different stress states are deduced, and the a nonlinear strength reduction method for high rock slope is established according to the relationship between normal stress and shear stress of rock mass under the Hoke-Brown criterion. The numerical calculation results show that the factor of safety (FOS) for high rock slope calculated by the proposed method is more reasonable. Taking the southwest slope of Dagushan Iron Mine as the research background, the safety factors of high rock slope under different rainfall conditions are calculated by COMSOL Multiphysics. And the stability analysis of high rock slope in open-pit mine under rainfall are carried out.