As a 3D topological insulator, bismuth selenide (Bi2Se3) has potential applications for electrically and optically controllable magnetic and optoelectronic devices. Understanding the coupling with its topological phase requires studying the interactions of carriers with the lattice on time scales down to the sub-picosecond regime. Here we use an X-ray free-electron laser to perform time-resolved diffraction to investigate the ultrafast carrier-induced lattice contractions and interlayer modulations in Bi2Se3 thin films. The lattice contraction depends on the carrier concentration and is followed by an interlayer expansion accompanied by oscillations. Using density functional theory (DFT) and the Lifshitz model, the initial contraction can be explained by van der Waals force modulation of the confined free carrier layers. Band inversion, related to a topological phase transition, is modulated by the expansion of the interlayer distance. These results provide insight into instantaneous topological phases on ultrafast timescales.