Chitinase-3-like protein-1 (YKL-40) is recognized as a novel neuroinflammatory biomarker related to neurodegenerative diseases. We aimed to explore whether serum YKL-40 can serve as a potential biomarker in patients with cerebral small vessel disease (CSVD), and possible neural correlates with white matter fiber damage and cognitive impairment. Blood samples were collected from 100 CSVD patients and 80 healthy controls (HCs) to measure serum YKL-40 levels. All subjects underwent diffusion tensor imaging (DTI) scanning and cognitive function assessment, and the CSVD group was classified into CSVD-no cognitive impairment (CSVD-NCI) and CSVD-mild cognitive impairment (CSVD-MCI) two subgroups. Binary logistic regression analysis was conducted to determine the independent risk factors for CSVD and CSVD-MCI. Receiver operating characteristic analysis was performed to assess the diagnostic value of YKL-40. Partial correlation analysis was carried out to explore the associations of DTI indices with YKL-40 and cognition. The serum YKL-40 levels of CSVD were significantly higher than those of the HCs, and the CSVD-MCI was higher than that in HCs and CSVD-NCI. YKL-40 was further identified as an independent risk factor for CSVD and CSVD-MCI, in addition, serum YKL-40 provided high diagnostic accuracy for CSVD and CSVD-MCI. Comparison of DTI indices showed that patients with CSVD accompanied by a significant decrease in white matter fibers integrity. Moreover, the altered DTI indices were significantly correlated with the elevated level of YKL-40 and worse cognitive performance. Our study suggests that YKL-40 may be a biomarker of CSVD and may cause cognitive impairment by disrupting white matter fiber integrity.