Phononic engineering at gigahertz (GHz) frequencies form the foundation of microwave acoustic filters, high-speed acousto-optic modulators, and quantum transducers. Terahertz (THz) phononic engineering could lead to acoustic filters and modulators at higher bandwidth and speed, as well as quantum circuits operating at higher temperatures. It can also enable new ways to manipulate and control thermal transport, as acoustic phonons at THz frequencies are the main heat carriers in nonmetallic solids. Despite its potential, methods for engineering THz phonons have been little explored experimentally due to the challenges of achieving the required material control at sub-nanometer precision and efficient phonon coupling at THz frequencies. Here, we demonstrate efficient generation, detection, and manipulation of THz phonons through precise integration of atomically thin layers in van der Waals heterostructures. We employ few-layer graphene (FLG) as an ultrabroadband phonon transducer, converting femtosecond near-infrared (NIR) pulses to broadband acoustic phonon pulses with spectral content up to 3 THz. A single layer of WSe2 is used as a sensor, where high-fidelity readout is enabled by the exciton-phonon coupling and strong light-matter interactions at the monolayer limit. By combining these capabilities in a single van der Waals heterostructure and detecting responses to incident mechanical waves, we performed THz phononic spectroscopy, similar to conventional optical spectroscopy which detects responses to incident electromagnetic waves. Using this platform, we demonstrate high-Q THz phononic cavities using hexagonal boron nitride (hBN) stacks. We further show that a single layer of WSe2 embedded in hBN can efficiently block the transmission of THz phonons. By comparing our measurements to a nanomechanical model, we obtain the important force constants at the hBN-graphene and hBN-WSe2 heterointerfaces. Our results could enable THz phononic metamaterials based on van der Waals heterostructures for ultrabroadband acoustic filters and modulators, as well as novel routes for thermal engineering.