This work reports the thermal conductivity of RbV3Sb5 and CsV3Sb5 with three-dimensional charge density wave phase transitions from 80 K to 400 K measured by pump-probe thermoreflectance techniques. At room temperature, the in-plane (basal plane) thermal conductivities are found moderate, i.e., 12 W m-1 K-1 of RbV3Sb5 and 8.8 W m-1 K-1 of CsV3Sb5, and ultralow cross-plane (stacking direction) thermal conductivities are observed, with 0.72 W m-1 K-1 of RbV3Sb5 and 0.49 W m-1 K-1 of CsV3Sb5 at 300 K. A unique glass-like temperature dependence in the cross-plane thermal conductivity is discovered, which decreases monotonically even lower than the Cahill-Pohl limit as the temperature decreases below the phase transition point TCDW. This temperature dependence is found to obey the hopping transport picture. In addition, a peak in cross-plane thermal conductivity is observed at TCDW as a fingerprint of the modulated structural distortion along the stacking direction.