The visual word form area (VWFA) plays a significant role in the development of reading skills. However, the developmental course and anatomical properties of the VWFA have only limitedly been investigated. The aim of the current longitudinal MRI study was to investigate dynamic, bidirectional relations between reading and the structure of the left fusiform gyrus at the early-to-advanced reading stage. More specifically, by means of bivariate correlations and a cross-lagged panel model (CLPM), the interrelations between the size of the left fusiform gyrus and reading skills (a composite score of a word and pseudo-word reading task) were studied in a longitudinal cohort of 43 Flemish children (29M, 14F) with variable reading skills in grade 2 (the early stage of reading) and grade 5 (the advanced stage of reading) of primary school. Results revealed that better reading skills at grade 2 lead to a larger size of the left fusiform gyrus at grade 5, whereas there are no directional effects between the size of the left fusiform gyrus at grade 2 and reading skills at grade 5. Hence, according to our results there is behavior-driven brain plasticity and no brain-driven reading change between the early and advanced stage of reading. Together with pre-reading brain studies showing predictive relations to later reading scores, our results suggest that the direction of brain-behavioral influences changes throughout the course of reading development.