The question of how much sleep is best for the brain attracts scientific and public interest, and there is concern that insuficient sleep leads to poorer brain health. However, it is unknown how much sleep is sufficient and how much is too much. We analyzed 51,295 brain magnetic resonnance images from 47,039 participants, and calculated the self-reported sleep duration associated with the largest regional volumes and smallest ventricles relative to intracranial volume (ICV) and thickest cortex. 6.8 hours of sleep was associated with the most favorable brain outcome overall. Critical values, defined by 95% confidence intervals, were 5.7 and 7.9 hours. There was regional variation, with for instance the hippocampus showing largest volume at 6.3 hours. Moderately long sleep (> 8 hours) was more strongly associated with smaller relative volumes, thinner cortex and larger ventricles than even very short sleep (< 5 hours), but effect sizes were modest. People with larger ICV reported longer sleep (7.5 hours), so not correcting for ICV yielded longer durations associated with maximal volume. Controlling for socioeconomic status, body mass index and depression symptoms did not alter the associations. Genetic analyses showed that genes related to longer sleep in short sleepers were related to shorter sleep in long sleepers. This may indicate a genetically controlled homeostatic regulation of sleep duration. Mendelian randomization analyses did not suggest sleep duration to have a causal impact on brain structure in the analyzed datasets. The findings challenge the notion that habitual short sleep is negatively related to brain structure.