A damping strategy using a friction ring damper for an industrial flywheel was numerically and experimentally investigated. The friction ring damper, located on the arms of the flywheel, was experimentally found to effectively reduce the vibration amplitude of the flywheel. The vibration energy is dissipated when relative motions occur at the friction contact interfaces. Nonlinear dynamic analysis based on a lumped-parameter model of a flywheel equipped with a friction ring damper was conducted. A dimensionless parameter, κ, defined as the ratio of the critical friction force to the amplitude of harmonic force, was used to evaluate the damping performance. For several values of κ, steady-state responses under harmonic excitation and nonlinear modes were obtained using the harmonic balance method (HBM) combined with the alternating frequency–time domain method (AFT). The forced response analysis proved the existence of an optimal value of κ, which can minimize the vibration amplitude of the flywheel. The nonlinear modal analysis showed that all the damping ratio–frequency curves are completely coincident even for different κ, and the frequency corresponding to the maximum damping ratio is equal to the frequency at the intersection of the forced response curves under the fully slip and the fully stick states of the friction contact interface. By analyzing the behaviors of the friction contact interface, it is shown that the friction contact interface provides damping in the combined stick–slip state. The forced response under random excitation was calculated using the Runge–Kutta method and the friction interface behaviors were analyzed. Finally, spectral testing was conducted to verify the numerical results.