The paper presents the results of a comprehensive statistical study on the optimization of the optical system for independent amplitude and phase wavefront manipulation, which is implemented on a binary digital micromirror device. Based on a statistical analysis of the data, an algorithm for selecting parameters (binary pattern carrier frequency, and aperture for first-order diffraction filtering) that ensures the optimal quality of the modulated wavefront was developed. This algorithm takes into account the type of modulation, i.e. amplitude, phase, or amplitude-phase type, the size of the coded distribution, and its requirements for spatial resolution and quantization. The results of this study will greatly contribute to modulated wavefront quality improvement in various applications with different requirements to spatial resolution and quantization.