Having experienced stress during sensitive periods of brain development strongly impacts how individuals cope with later stress. Many become more prone to develop anxiety or depression, but some appear resilient. The as-yet-unknown mechanisms underlying these differences may lie in how genes and environmental stressors interact to shape the circuits controlling emotions. Here, we investigated the role of the habenulo-interpeduncular system (HIPS), a critical node of reward circuits, in early stress-induced anxiety in mice. Based on immediate early gene expression, we found that a subcircuit of this system, characterized by Otx2 expression, is particularly sensitive to chronic restraint stress during the peripubertal period, and that this induces hypersensitivity of the HIPS to later stress and susceptibility to develop anxiety. We also show that conditional knockout of Otx2 restricted to the HIPS in mice counteracts these effects of stress. Together, these results demonstrate that a genetic factor, Otx2, and stress interact around puberty to shape the stress sensitivity of the HIPS, revealing this sensitivity as a key modulator of susceptibility/resilience to develop anxiety.