As promising therapeutic targets for several types of cancers, Aurora kinases A and B share remarkably high sequence and structure similarity, which render the design of subtype selective small-molecule inhibitors challenging. Since Aurora A acts as both an oncogene and a haploinsufficient tumor suppressor, inhibiting Aurora B with highly selective inhibitors represents a less toxic anti-cancer strategy. However, the molecular mechanism governing the ligand selectivity for Aurora A over B reminds elusive. In this study, an experimentally validated ligand, Barasertib which displays 1000-fold selectivity for Aurora B over A was used as a template molecule, and the selectivity mechanism was investigated by molecular dynamics simulation and binding free energy analyses. The computed binding free energies are consistent with the experimental bioactivity of Barasertib. The major residues that contribute to ligand binding were also identified. The hinge residue Arg159, only found in Aurora B was found to contribute significantly to the binding of Aurora B, while no such binding was found in Aurora A. A highly unfavorable polar interaction with Glu181 in Aurora A hinders the ligands binding ability. These 2 interactions play a key role in the observed selectivity of Barasertib for Aurora B over A. Interestingly, the binding of partner proteins forces the aC helix and the beta sheets in the N-lobe to move inward towards the ATP binding pocket, leading to a smaller hydrophobic back pocket and unfavorable binding. Taken together, our results show how differential behavior of the ligand arises from a complex interplay of small but relevant structural differences upon binding to different kinase subtypes. The insights gained into the structural determinants of subtype selectivity will facilitate the further development of highly selective and potent inhibitors as drug candidates for cancer therapy.