Recent whole brain mapping projects are collecting large-scale 3D images using powerful and informative modalities, such as STPT, fMOST, VISoR, or MRI. Registration of these multi-dimensional whole-brain images onto a standard atlas is essential for characterizing neuron types and constructing brain wiring diagrams. However, cross-modality image registration is challenging due to intrinsic variations of brain anatomy and artifacts resulted from different sample preparation methods and imaging modalities. We introduced a cross-modality registration method, called mBrainAligner, which uses coherent landmark mapping as well as deep neural networks to align whole mouse brain images to the standard Allen Common Coordinate Framework atlas. We also built a single cell resolution atlas using the fMOST modality, and used our method to generate whole brain map of 3D full single neuron morphology and neuron cell types.