Water quality is the restrictive factor for both ecosystem health and social development in the Chinese Loess Plateau, a unique area with most severe soil erosion, fragile ecology, and water shortage. Understanding the characteristics of the pollutant loads is of vital importance for the sustainability of eco-environment in the Loess Plateau. This study investigated the spatiotemporal changes of chemical oxygen demand (COD), total nitrogen (TN), and total phosphorus (TP) loads by combining the Soil and Water Assessment Tool (SWAT) and regression model Load Estimator (LOADEST) in a typical loess hilly and gully watershed—the Yan River Basin (YanRB). Results showed that the model simulations of monthly streamflow and pollutant loads were in good agreement with those derived from the in-situ observations. The temporal variation analysis suggested that the pollutant loads were generally rising in the study period (2001–2018) at four of the five stations and reached the maximum in 2014, and the multi-year (i.e., 2001–2018 with 2013 being excluded due to extreme rainfall) average loads of COD, TN, and TP at the Tanjiahe station, which is close to the outlet of the basin, were 15,021 kg/d, 3,835 kg/d, and 168 kg/d, respectively. The spatial distribution of the TN and TP loads along the river seemed to be quite unique because the TP level were obviously higher at the midstream (e.g., Zhujiagou and Ganguyi) than the downstream (e.g., Tanjiahe), and the TN level decreased when the river flowed from Zhujiagou to Ganguyi. Further, the seasonal analysis indicated that the nutrient loads were the highest in summer, followed by autumn, and the loads in these two wet seasons contributed the most of the annual pollution loads—about 76% and 84% for TN and TP, respectively, indicating the higher flow, the higher pollution load, a similar point based on the inter-annual analysis. In addition, the contribution analysis of point source and non-point source pollutions demonstrated that NPS led to most of the pollutant loads at the whole watershed—87%, 85%, and 84% of the COD, TN, and TP loads, respectively. Overall, this study provided spatiotemporal distributions of the key pollutant loads in the YanRB and can be valuable for water quality protection and pollution control in this area.