Could human cooperation be the result of evolution favouring groups where altruists punish non-altruists 1-4? While punishment can benefit groups through increased cooperation, it often costs more than it is worth, challenging group level explanations 1,5,6. We therefore tested if altruistic punishment could spread among groups via imitation (social learning) rather than direct competition. Groups observed either how punishment increased cooperation, or how it also decreased profits, in a previous experiment. Individuals then voted to either enable or disable all punishment. We found (1) that punishment was common, but most individuals voted to disable punishment and (2) observing how punishment affected payoffs drastically disfavoured punishment, and consequently cooperation. Our results contradict the hypothesis that humans have evolved to cooperate via cultural group selection favouring altruistic punishment 2,3,7-11.