Intelligence is a strong predictor of occupational achievement, quality of life and physical health. While variation in intelligence is strongly heritable and has been robustly associated with early environment and brain morphology, little is known about how these factors combine and interact to explain this variation in intelligence. To address this, we modelled the relationship between common genetic variation, grey matter volume, early life adversity and education and intelligence in a UK Biobank sample of 16,383 individuals using structural equation modelling. We tested the hypotheses that total grey matter volume would mediate the association between genetics and intelligence, and that early life adversity and educational attainment would moderate this relationship. Common genetic variation was estimated using a genome-wide polygenic score (PGS) of intelligence. Consistent with previous studies, we found that grey matter volume was a significant mediator of the relationship between PGS and intelligence, but only when early life adversity was included in the model. We further found that higher educational attainment in turn moderated the effects of adversity. These findings suggest an important interaction between genetic and environmental factors in the development of intellectual functioning.