It is commonly known that the dephasing in open quantum systems is due to the establishment of bipartite correlations with an environment. Recently, a new approach of average over disordered Hamiltonian ensemble is developed and shown to capable of describing both the incoherent dynamical behavior and the nonclassicality of dynamical processes. Here we further extend the approach of Hamiltonian ensemble in the canonical form to the realm of structural disorder. Under the separation of the probability distribution within the Hamiltonian ensemble, the geometrical structural is easily visualized and can be characterized according to the degree of symmetry. We demonstrate four degrees and investigate the effects of different types of symmetry breaking on the incoherent dynamics. With these effects, we obtain rather general master equations, going beyond the previous frameworks of pure dephasing or isotropic depolarization. The practicality of the Hamiltonian ensemble and the theory of process nonclassicality is significantly enhanced.