China’s industrial water withdrawal soared in the last decades and remained high. Stringent water management policies were set to save water through improving industrial withdrawal efficiency by 20% between 2015 and 2020. Although China has a nation-wide water scarcity, scarcity at city-level has not been fully explored. Thus, it is meaningful to use sectoral data to investigate industrial water saving potential and implication for alleviating scarcity. Here, we account for water withdrawal and scarcity in 272 prefectural cities, using a 2015 data benchmark. The top 10% of low-efficiency sectors occupied 46% water use. In scenario analysis of 41 sectors across 146 water scarce cities, we assume a convergence of below-average efficiencies to the national sector-average. Results reveal overall efficiency could be increased by 20%, with 18.9 km3 (±3.2%) water savings, equivalent to annual water demand of Australia or Hebei province in China. A minority of sectors (13%) could contribute to most (43%) water savings whilst minimizing economic perturbations. In contrast, implementing water efficiency measures in the majority of sectors would result in significant economic disruption to achieve identical savings. Water efficiency improvements should be targeted towards this minority of sectors: cloth(ing) supply-chain, chemical manufacturing, and electricity and heat supply.