Aims/hypothesis In Japan, the disastrous effects of the 2011 Tohoku Earthquake has affected aseismic design. Furthermore, the need for dynamic nonlinear analysis in the aseismic evaluation of essential structures such as nuclear power plants has increased. This study conducted cyclic shear box tests using rock mass samples to verify the nonlinear model used for the dynamic analysis. The shear load–shear displacement relationship and the maximum shear strength exerted at a small shear displacement level could be evaluated by comparing the outcomes of the existing nonlinear model with the experimental results. However, the shear load–shear displacement relationship at a large shear displacement level could not be accurately reproduced. Therefore, we modeled the damping characteristics of rocks after failure based on the results of laboratory tests on rock discontinuities, and improved the existing nonlinear model. Using the improved model, we could accurately reproduce the load–displacement relationship, rock internal strain, and rock boundary stress at a wide range of shear displacement levels.