Driven by access to large volumes of detailed movement data, the study of human mobility has grown rapidly over the past decade. This body of work has argued that human mobility is scale-free, it has proposed models to generate scale-free moving distance distributions, and explained how the scale-free distribution arises from aggregating displacements across scales. The field of human mobility, however, has not explicitly addressed how mobility is structured by the constraints set by geography -- that mobility is shaped by the outlines of landmasses, lakes, and rivers; by the placement of buildings, roadways, and cities. Based on unique datasets capturing millions of moves between precise locations, we show how separating the effect of geography from mobility choices, reveals a universal power law spanning five orders of magnitude (10m to 1,000,000m). To do so we incorporate geography via the `pair distribution function', a fundamental quantity from condensed matter physics that encapsulates the structure of locations on which mobility occurs. We show that this distribution captures the constraints that geography places on human mobility across length scales. Our description conclusively addresses debates between distance-based and opportunity-based perspectives on human mobility: By showing how the spatial distribution of human settlements shapes human mobility, we provide a novel perspective that bridges the gap between these previously opposing ideas.