It is generally believed that long-duration gamma-ray bursts (GRBs) are associated with massive star core-collapse, whereas short-duration GRBs are associated with mergers of compact star binaries. However, growing observations have suggested that oddball GRBs do exist, and multiple criteria (prompt emission properties, supernova/kilonova associations, and host galaxy properties) rather than burst duration only are needed to classify GRBs physically. A previously reported long-duration burst, GRB 060614, could be viewed as a short GRB with extended emission if it were observed at a larger distance and was associated with a kilonova-like feature. As a result, it belongs to the Type-I (compact star merger) GRB category and is likely of the binary neutron star merger origin. Here we report a peculiar long-duration gamma-ray burst, GRB 211211A, whose prompt emission properties in many aspects differ from all known Type-I GRBs, yet its multi-band observations suggest a non-massive-star origin. In particular, significant excess emission in both optical and near-infrared wavelengths has been discovered, which resembles kilonova emission as observed in some Type-I GRBs. These observations point towards a new progenitor type of GRBs. A scenario invoking a white dwarf-neutron star merger with a post-merger magnetar engine provides a self-consistent interpretation for all the observations, including prompt gamma-rays, early X-ray afterglow, as well as the engine-fed kilonova emission.