Backlash is one of main nonlinear internal excitation factors in gear transmission system and therefore has been widely concerned. Most existing models of backlash are based on a random constant, which ignore the dynamic characteristics of backlash itself and the effects of surface topography. To model the backlash precisely, in this paper, the constant part of backlash is revised through average height of all asperities in contact region related to surface roughness by fractal method. Simultaneously, the dynamic part is modeled considering the displacement of gear center motion that comes from shaft deformation in coupling dynamic meshing. A complete backlash model consisting of the two parts is established subsequently and a corresponding close-loop algorithm is proposed to solve system dynamics by coupling mesh stiffness and time varying pressure angle. Through time history charts, phase portraits and Poincare mapping as well as frequency spectrograms, calculation results clearly demonstrate the comprehensive effects of dynamic backlash on the nonlinear dynamics involving vibration amplitude, frequency and chaotic characteristics of a spur gear pair. The effects of surface topography on backlash and system nonlinear response including vibration amplitude and chaotic features are also analyzed, therefore dynamic backlash and surface topography are important factors that cannot be ignored in gear issue. The comparison with experimental data as well as other previous models is conducted to verify the superiority of proposed model.