Polar Rashba-type semiconductor BiTeI doped with magnetic elements constitutes one of the most promising platforms for the future development of spintronics and quantum computing thanks to the combination of strong spin-orbit coupling and internal ferromagnetic ordering. The latter originates from magnetic impurities and is able to open an energy gap at the Kramers point (KP gap) of the Rashba bands. In the current work using angle-resolved photoemission spectroscopy (ARPES) we show that the KP gap depends non-monotonically on the doping level in case of V-doped BiTeI. We observe that the gap increases with V concentration until it reaches 3% and then starts to mitigate. Moreover, we find that the saturation magnetisation of samples under applied magnetic field studied by superconducting quantum interference device (SQUID) magnetometer has a similar behaviour with the doping level. Theoretical analysis shows that the non-monotonic behavior can be explained by the increase of antiferromagnetic coupled atoms of magnetic impurity above a certain doping level. This leads to the reduction of the total magnetic moment in the domains and thus to the mitigation of the KP gap as observed in the experiment. These findings provide further insight in the creation of internal magnetic ordering and consequent KP gap opening in magnetically-doped Rashba-type semiconductors.