The winding system is a time-varying system that takes into account various complex nonlinear characteristics. Among them, how to control the stability of the winding tension during the winding process is the primary problem that has plagued the development of this field in recent years. There are many nonlinear factors that affect the tension in the winding process, such as friction, structured, unstructured uncertainties and external interference. These terms severely restrict the performance of the tracking error of tension. Previously, the existing tension control strategies were mainly based on the composite control of the tension loop and the speed loop, and the research was carried out through complex decoupling operations. Due to the large calculation amount of this method, it brings a lot of inconvenience in practical engineering applications. For this purpose that further simplify the tension generation mechanism and the influence of the nonlinear characteristics of the winding system, a simpler nonlinear dynamic model of the winding tension was established. At the same time, an adaptive method was applied to update the feedback gain of the continuous robust integral of the sign of the error (RISE). Furthermore, extended state observer (ESO) was used to estimate the modeling errors and external disturbances. Then, the model disturbance term can be compensated in the designed controller of RISE. The asymptotic stability of the system is prove based on the stability theory of Lyapunov. Finally, a comparative analysis was carried out between the proposed nonlinear controller and several other controllers. Results show that the tracking error of winding tension was enhanced significantly.