Increasing global food demand will require more food production without further exceeding the planetary boundaries, while at the same time adapting to climate change. We used an ensemble of wheat simulation models, with sink-source improved traits from the highest-yielding wheat genotypes to quantify potential yield gains and associated N requirements. This was explored for current and climate change scenarios across representative sites of major world wheat producing regions. The sink-source traits emerged as climate neutral with 16% yield increase with current N fertilizer applications under both current climate and mid-century climate change scenarios. To achieve the full yield potential, a 52% increase in global average yield under a mid-century RCP8.5 climate scenario, fertilizer use would need to increase fourfold over current use, which would unavoidably lead to higher environmental impacts from wheat production. Our results show the need to improve soil N availability and N use efficiency, along with yield potential.