The Moon holds important clues to the early history of the Solar System. The large scars evident on its near side are impact basins filled with dark basalt; some 50 lunar basins (crater diameter $D>300$ km) have been recognized overall$^1$. The question of the origin of basin-forming impactors remained unsettled. Here we show that most impactors were rocky planetesimals left behind at $\sim 0.5$--$1.5$ au after the terrestrial planet accretion. The number of basins expected from impacts of leftover planetesimals largely exceeds the number of known lunar basins, suggesting that the first $\sim200$ Myr of impacts is not recorded on the lunar surface$^{2,3}$. The Imbrium basin formation (age$^4$ $\simeq 3.92$~Gyr; impactor diameter$^{5,6}$ $d>100$ km) occurs with a 15--35% probability in the model. Imbrium must have formed unusually late to have only two smaller basins (Orientale and Schr"odinger) forming afterwards. The model predicts $\simeq 20$ $d>10$-km impacts on the Earth 2.5--3.5 Gyr ago (Ga), which is comparable to the number of known spherule beds in the late Archean$^7$.