Many animal feeds compete for resources with human food production. The current use of food system by-products and residues as feed could potentially be increased to reduce the competition. We gathered a harmonised global food system material flow database for crop, livestock and aquaculture production including the availability of food system by-products. This allowed us to analyse the potential to replace the food-competing feedstuff, here cereals, whole fish, vegetable oils and pulses, that currently account for 11% of total feed used globally, with available food system by-products. While considering the nutritional requirements in animal production, we found that the replacement could free food-grade feeds for human consumption and increase the current food supply in kcal by 11-17% (6-11% if the use of crop residues is not accounted for) and in terms of protein 11-15% (9-14%). Our results thus indicate that the increased feed use of by-products has considerable potential, particularly when used in combination with other measures, in the much needed transition towards more sustainable and circular food systems.