Nonlinear Quasi-zero-stiffness (QZS) vibration isolation systems with linear damping cannot lead to displacement isolation with different excitation levels. In this study, a QZS system with nonlinear hysteretic damping was investigated. The Duffing-Ueda equation with a coupling nonlinear parameter 𝜂 was proposed to describe the dynamic motion of the QZS system. By using the harmonic balance method (HBM), the primary and secondary harmonic responses were obtained and verified by numerical simulations. The results indicated that nonlinear damping can guarantee a bounded response for different excitation levels. The one-third subharmonic response was found to affect the isolation frequency range even when the primary response was stable. To evaluate the performance of the QZS system, the effective isolation frequency Ω𝑒 and maximum transmissibility 𝑇𝑝 were proposed to represent the vibration isolation range and isolation effect, respectively. By discussing the effect of 𝜂 on Ω𝑒 and 𝑇𝑝, the conditions to avoid nonlinear phenomena and improve the isolation performance are provided. A prototype of the QZS system was then constructed for vibration tests, which verified the theoretical analysis.