To reduce the adverse vibrations of buildings, tuned mass dampers (TMDs), which are the most representative passive control devices, have been widely used and studied in aerospace, machinery, civil engineering, and other fields for many years. Most scholars used to treat the TMD as a linear damper, but they show some nonlinear characteristics owing to the use of limit devices and large displacements. It is necessary to consider the nonlinear coefficient of the TMD when designing its parameters. In this study, the mass ratio of the TMD was optimized with considering the nonlinear coefficient of the TMD. The complex variable average method and multiscale method were used for analysis. A mass ratio interval was found on the “ε- N2” curve in which modulation response can occur, and then an analytical method for obtaining the optimal mass ratio of TMD was derived based on this phenomenon. The numerical results showed that taking the midpoint of this mass ratio interval as the optimal mass ratio can yield a better damping effect and robustness than using the traditional linear design method.