Teeth disengaging or back-side teeth meshing induced by backlash reduces the transmission quality and dynamic performance of gear systems, and accurate interpretation of multi-state meshing behavior can provide guidance for structural optimization and performance evaluation. Therefore, the multi-state meshing behavior of the gear system is elaborated. A new nonlinear dynamic model of a spur gear system with five-state meshing behavior is proposed based on time-varying backlash and contact ratio. The time-varying meshing stiffness and time-varying backlash considering the elastic contact of gear teeth, gear temperature rise and lubrication are included in the model. The five-state meshing behavior is clearly characterized by constructing five Poincaré maps, and its generation mechanism is studied using dynamic meshing force time history, teeth relative displacement time history and phase portrait. The bifurcation and evolution of five-state meshing behavior are analyzed under the effects of load factor, meshing frequency and error coefficient. The results show that the mutation in the direction of dynamic meshing force leads to teeth disengaging and back-side single or double teeth contact, forming multi-state meshing behavior. Bifurcation caused by parameter changes greatly affects the evolution of five-state meshing behavior, particularly grazing bifurcation can decrease the number of teeth disengagement. Chaotic behavior or trajectory expansion inspires multi-state meshing vibration of the system. Previous gear system models could not reveal these phenomena due to ignoring the multi-state meshing behavior.