Programmable optical circuits form a key part of quantum technologies today, ranging from transceivers for quantum communication to integrated photonic chips for quantum information processing. As the size of such circuits is increased, maintaining precise control over every individual component becomes challenging, leading to a reduction in the quality of the operations performed. In parallel, minor imperfections in circuit fabrication are amplified in this regime, dramatically inhibiting their performance. Here we show how embedding an optical circuit in the higher-dimensional space of a large, ambient mode-mixer using inverse-design techniques allows us to forgo control over each individual circuit element, while retaining a high degree of programmability over the circuit. Using this approach, we implement high-dimensional linear optical circuits within a complex scattering medium consisting of a commercial multi-mode fibre placed between two controllable phase planes. We employ these circuits to manipulate high-dimensional spatial-mode entanglement in up to seven dimensions, demonstrating their application as fully programmable quantum gates. Furthermore, we show how their programmability allows us to turn the multi-mode fibre itself into a generalised multi-outcome measurement device, allowing us to both transport and certify entanglement within the transmission channel. Finally, we discuss the scalability of our approach, numerically showing how a high circuit fidelity can be achieved with a low circuit depth by harnessing the resource of a high-dimensional mode-mixer. Our work serves as an alternative yet powerful approach for realising precise control over high-dimensional quantum states of light, with clear applications in next-generation quantum communication and computing technologies.