Artemisia trifida (giant ragweed) is an invasive weed with an expanding distribution area. In recent years it has been found to invade grasslands, bringing great challenges for effective weed control and restoration of native herbage. Although it has been reported that plant invasion can cause a decline in species richness and biodiversity in native seed banks, which may eventually lead to the depletion of native seed banks, few location- and species-specific case studies have been conducted regarding the dynamic characteristics of the invaded seed banks from invasion back to restoration. The purpose of this study was to compare and quantify the seed banks of grassland communities after (1) giant ragweed invasion for 0-8 years, and (2) giant ragweed removal, in Yili Valley, Xinjiang, China. The results showed that the duration of invasion determined whether giant ragweed could pose a significant threat to the native community seed bank. The seed bank density of native community had significantly decreased by 30.44% after 4 years of invasion, and in the sixth year, the species richness in the seed bank had decreased significantly by 12.36%. After the invasion had lasted for eight years, the seed bank density of the native community had decreased by 83.28%, the species richness in the seed bank decreased by 39.33%, and the seed bank tended to be homogeneous. After the giant ragweed was removed, the potential for restoration was limited by the residual seed bank. Three years after the restoration, although the density of seed banks increased significantly, new growth was dominated by weedy species, rather than crucial components of the grassland habitat. This study is of great significance to the control of giant ragweed and the restoration of grassland vegetation invaded by giant ragweed.