Defect centers in hexagonal boron nitride (hBN) have been extensively studied as room-temperature single-photon sources. The electronic structure of these defects exhibits strong coupling to phonons, as evidenced by the observation of phonon sidebands in both photoluminescence and cathodoluminescence spectra. However, the dynamics of the electron-phonon coupling as well as phonon-mediated dephasing of the color centers in hBN remain unexplored. Here, we apply a novel time-resolved CL spectroscopy technique (Nature Physics 19, 869–876 (2023)) to explore the population decay to phonon states and the dephasing time T2 with sub-femtosecond time resolution. We demonstrate an ultrafast dephasing time of only 200 fs and a radiative decay of about 585 fs at room temperature, in contrast with all-optical time-resolved photoluminescence techniques that report a decay of a few nanoseconds. This behavior is attributed to efficient electron-beam excitation of coherent phonon-polaritons in hBN, resulting in faster dephasing of electronic transitions. Our results demonstrate the capability of our sequential cathodoluminescence spectroscopy technique to probe the ultrafast dephasing time of single emitters in quantum materials with sub-femtosecond time resolution, heralding access to quantum-path interferences in single emitters coupled to their complex environment.