The genetic underpinnings of cognitive resilience in aging remains unknown. Predicting an individual’s rate of cognitive decline—or cognitive resilience—using genetics will allow personalized intervention for cognitive enhancement and optimal selection of target samples in clinical trials. Here, using genome-wide polygenic scores(GPS) as the genomic indicators for variations of human intelligence, we examined the genetic liability of cognitive abilities in the behavioral/cognitive phenome to understand individual differences in cognitive capacity over time. Using the longitudinal sociogenomic data of 8,509 European-ancestry adults between the ages of mid-60s to 70s, we found that a higher cognitive GPS significantly correlated with a slower cognitive decline specifically in memory recall, but not in other cognitive domains. Linear mixed models with cognitive GPSs explained proportions of the variances in cognitive tests up to 60.4%. This study presents the novel genetic protective effects of cognitive ability on the decline of memory recall in aging population.