Following a disaster, crucial decisions about recovery resources often focus on immediate impact, partly due to a lack of detailed information on who will struggle to recover. Here we perform an analysis of surveyed data on reconstruction and secondary data commonly available after a disaster to estimate a metric of non-recovery or the probability that a household could not fully reconstruct within five years after an earthquake. Analyzing data from the 2015 Nepal earthquake, we find that non-recovery is associated with a wide range of factors beyond building damage, such as ongoing risks, population density, and remoteness. If such information were available after the 2015 earthquake, it would have highlighted that many damaged areas have differential abilities to reconstruct due to these factors. More generally, moving beyond damage data to evaluate and quantify non-recovery will support effective post-disaster decisions that consider pre-existing differences in the ability to recover.