Brain age can be estimated using different MRI modalities including diffusion MRI. Recent studies demonstrated that white matter (WM) tracts that share the same function might experience similar alterations. Therefore, in this work, we sought to investigate such issue focusing on five WM bundles holding that feature that is Association, Brainstem, Commissural, Limbic and Projection fibers, respectively. For each tract group, we estimated brain age for 15335 healthy participants from United Kingdom Biobank relying on diffusion MRI image derived endophenotypes, Bayesian ridge regression modeling and 10 fold-cross validation. Furthermore, we estimated brain age for an ensemble model that gathers all the considered white matter bundles. Association analysis was subsequently performed between the estimated brain age delta as resulting from the six models, that is for each tract group as well as for the ensemble model, and 38 daily life style measures, 14 cardiac risk factors and cardiovascular magnetic resonance imaging features and genetic variants. Our study revealed that the limbic tracts experience less brain aging compared to other tract groups, Brainstem tracts ages relatively faster while the other tract groups present similar brain aging patterns. The model achieved 5.86 mean absolute error (MAE) for brainstem, 5.08 for limbic tracts and around 5.2 for the other three tract groups while, the ensemble model 4.55. Moreover, the results suggest that limbic tracts are more affected by daily lifestyle factors than the other tract groups. Lastly, two SNPs were significantly (p-value < 5E-8) associated with brain age delta in projection fibers. Those SNPs are mapped to HIST1H1A and SLC17A3 genes.