Body mass index (BMI) is associated with cognitive abilities, but the nature of the relationship remains largely unexplored. We aimed to investigate the bidirectional relationship from midlife through late-life, while considering sex differences and genetic predisposition to higher BMI. We used data from 23,892 individuals of European ancestry from the Health and Retirement Study, with longitudinal data on BMI and three established cognitive indices: mental status, episodic memory, and their sum, called total cognition. To investigate the dynamic relationship between BMI and cognitive abilities, we applied dual change score models of change from age 50 through 89, with a breakpoint at age 65 or 70. Models were further stratified by sex and genetic predisposition to higher BMI using tertiles of a polygenic score for BMI (PGSBMI). We demonstrated bidirectional effects between BMI and all three cognitive indices, with higher BMI contributing to steeper decline in cognitive abilities in both midlife and late-life, and higher cognitive abilities contributing to less decline in BMI in late-life. The effects of BMI on change in cognitive abilities were more evident in men compared to women, and among those in the lowest tertile of the PGSBMI compared to those in the highest tertile, while the effects of cognition on BMI were similar across groups. In conclusion, these findings highlight a reciprocal relationship between BMI and cognitive abilities, indicating that the negative effects of a higher BMI persist from midlife through late-life, and that weight-loss in late-life may be driven by cognitive decline.