The Air Turbine Rocket (ATR) engine is a promising combined cycle propulsion engine. This paper uses the polynomial nonlinear state-space (PNLSS) model to model and identify the nonlinear system of the ATR engine. A method of multiple uncorrelated step signals is proposed as the excitation signals for the nonlinear system. The adaptive Nelder-Mead simplex (ANMS) algorithm is used as the nonlinear least squares optimization algorithm to solve the PNLSS model parameters. The identification results show that the multiple uncorrelated step signals have good excitation effects on the steady-state operating points and the large-scale dynamic processes of the nonlinear system. Under the same initial values, the ANMS algorithm has obvious advantages over the Levenberg-Marquardt (L-M) algorithm and the standard Nelder-Mead simplex (SNMS) algorithm in terms of optimization effect and convergence speed. The PNLSS model shows higher fitting accuracy and prediction ability than the linear state space (LSS) model for the operating points and the wide-range dynamic processes of the ATR engine. This study provides a new method for excitation signal design and parameter identification for nonlinear systems and lays a foundation for the design of nonlinear controllers.