Biomembranes experience out-of-equilibrium conditions in living cells. Their undulation spectra are different from those in thermal equilibrium. Here, we report on the undulation of a fluid membrane pushed by the stepwise growth of filaments as in the leading edge of migrating cells, using three-dimensional Monte Carlo simulations. The undulations are largely modified from equilibrium behavior. When the tension is constrained, the low-wave-number modes are suppressed or enhanced at small or large growth step sizes, respectively, for high membrane surface tensions. In contrast, they are always suppressed for the tensionless membrane , wherein the wave-number range of the suppression depends on the step size. When the membrane area is constrained, in addition to these features, a specific mode is excited for zero and low surface tensions. The reduction of the undulation first induces membrane buckling at the lowest wave-number, and subsequently, other modes are excited, leading to a steady state.