Habitat loss is the leading cause of global biodiversity decline, but the influence of human pressure within the matrix surrounding habitat fragments remains poorly understood. Here we measure the relationship between fragmentation, matrix condition (measured as the extent of high human footprint levels), and the change in extinction risk of 4,327 terrestrial mammals. We find that the matrix condition and the fragmentation of habitat are strongly associated with changes in species extinction risk. Importantly, we discover that fragmentation is a stronger predictor of risk than species life-history traits, habitat loss, and habitat amount. Moreover, the importance of fragmentation increases with an increasing deterioration of the matrix condition, highlighting the critical influence matrix quality plays on the effects of fragmentation. These findings suggest that restoration measures in habitat matrices may be an important conservation action for mitigating the effects of fragmentation relative to extinction risk of terrestrial mammals.