A woman’s risk of breast cancer risk can be assessed through models that integrate physiological, environmental and genetic factors. Here we use genome-wide data from 309,479 women including 128,771 with clinical data to build and validate ancestry-specific polygenic risk scores (PRS) for breast cancer (BC) and use them to explore the interplay between clinical factors and rare and common genetic variation on cancer risk. We show that integrating PRSs into the Tyrer-Cuzick risk model improves the classification of risk in prospective cohort data from more than 100,000 women. Across different genetic ancestry groups, we show that PRSs identify women at equivalent risk of cancer as carriers of pathogenic variants in key BC susceptibility genes, but who are more than 5 times more numerous at the population level. The PRSs substantially improve BC risk stratification in women with monogenic BC susceptibility syndromes, with women with the highest PRS values having around twice the risk of those with the lowest values. Women with high PRS values accrue per annum risk more quickly and at younger ages than those with average or low PRS values, leading to a greater than 20 year age discrepancy in reaching guideline-delineated high-risk thresholds, depending on their PRS value. These results have important implications for BC risk mitigation in diverse populations and advocate for the inclusion of PRSs in breast cancer prevention strategies.