Qubit memory that is entangled with photonic qubit is the building block for quantum repeaters. Realizing ensemble-based repeaters requires qubit simultaneously featuring high retrieval efficiencies and long lifetimes. So far, the longest memory time at 50% retrieval efficiency for entanglement storage reaches ~40 μs. Here, by coupling cold atoms to two spatially-distinctive modes of a polarization-interferometer-based cavity, we achieve cavity-perfectly-enhanced and long-lived atom-photon entanglement. A write-laser beam is applied onto cold atoms, we then create a magnetic-field-insensitive spin-wave qubit that is entangled with the photonic qubit encoded onto two arms of the interferometer. By applying a read beam, the spin-wave qubit is converted to a photonic qubit, whose two modes are resonant with the cavity. Our experimental data shows that zero-delay intrinsic retrieval efficiency is up to 77% and 1/e lifetime 1ms. At 50% efficiency, the storage time reaches 540μs , which is 13.5 times longer than the best reported result.