In this study, a new innovative hybrid membrane/vacuum swing adsorption (VSA) process is developed, modeled, and optimized for removal of CO2 from flue gases. The process benefits from the advantages of membrane simplicity and the high product quality of the adsorption system. The main advantage of this new process is the simultaneous increases of both CO2 purity and its recovery. To achieve this objective, in the first step, a membrane system using PEBAX nano-composite membrane was modeled. In the second step, a VSA system using Zeolite 13X was modeled. The adsorption equilibrium was predicted by the Toth isotherm. To increase the modeling accuracy, the mass transfer rate was calculated based on the quasi-second order model. At the final step, the hybrid membrane/VSA process was modeled. Comparison of the new hybrid membrane/VSA with the stand-alone VSA process shows that the CO2 product concentration was increased by 39% and the recovery was improved by 8%. To study the process limitations and increase the product quality, a sensitivity analysis was performed on vacuum pressure, membrane stage cut, and recycle ratio. Based on the results, decreasing the membrane stage cut to 15% and applying a recycle ratio equal to 2 will increase the product quality with the cost of increasing the equipment size. Finally, to achieve the required purity and recovery specification in industrial applications, the process was optimized using the genetic algorithm. Based on these results, it is possible to produce CO2 with 94.7% purity and 99% recovery and N2 with 99.9% purity and 97.3% recovery by regenerating the adsorbents at 0.01 bar, setting the membrane stage cut equal to 11%, keeping the recycle ratio at 1.89, and adjusting the purge to feed ratio to 2%.