Spin-orbit coupling effect in structures with broken inversion symmetry, known as the Rashba effect, facilitates spin-orbit torques (SOTs) in heavy metal/ferromagnet/oxide structures, along with the spin Hall effect. Electric-field control of the Rashba effect is established for semiconductor interfaces, but it is challenging in structures involving metals owing to the screening effect. Here, we report that the Rashba effect in Pt/Co/AlOx structures is laterally modulated by electric voltages, generating out-of-plane SOTs. This enables field-free switching of the perpendicular magnetization and electrical control of the switching polarity. Changing the gate oxide reverses the sign of out-of-plane SOT while maintaining the same sign of voltage-controlled magnetic anisotropy, which confirms the Rashba effect at the Co/oxide interface is a key ingredient of the electric-field modulation. The electrical control of SOT switching polarity in a reversible and non-volatile manner can be utilized for programmable logic operations in spintronic logic-in-memory devices.