Retinotopic map, the mapping between visual inputs on the retina and neuronal responses on cortical surface, is one of the central topics in vision science. Typically, human retinotopic maps are constructed by analyzing functional magnetic resonance responses to designed visual stimuli on cortical surface. Although it is widely used in visual neuroscience, retinotopic maps are limited by measurement noise and resolution. One promising approach to improve the quality of retinotopic maps is to register individual subject’s retinotopic maps to a retinotopic template or atlas. However, none of the existing retinotopic registration methods has explicitly quantified the diffeomorphic condition, that is, retinotopic maps can be aligned by stretching/compressing but without tearing up. Here, we developed Diffeomorphic Registration for Retinotopic Maps (DRRM) to simultaneously align retinotopic maps in multiple visual regions under the diffeomorphic condition. Specifically, we used the Beltrami coefficient to model the diffeomorphic condition and performed surface registration based on retinotopic coordinates. The overall framework is simple and elegant and preserves topological condition defined in the atlas. We further developed a unique performance evaluation protocol and compared the performance of the new method with several existing image intensity-based registration methods on both synthetic and real datasets. The results showed that DRRM is superior to the existing methods in achieving diffeomorphic mappings in synthetic and empirical data from 3T and 7T magnets. DRRM may improve the interpretation of low-quality retinotopic maps and facilitate adoption of retinotopic maps in clinical settings.