There are two ways to unify gravitational field and gauge field. One is to represent gravitational field as principal bundle connection, and the other is to represent gauge field as affine connection. Poincare gauge theory and metric-affine gauge theory adopt the first approach. This paper adopts the second. We show a generalization of Riemannian geometry and a new affine connection, and apply them to establishing a unified coordinate description of gauge field and gravitational field. It has the following advantages. (i) Gauge field and gravitational field have the same affine connection representation, and can be described by a unified spatial frame. (ii) Time can be regarded as the total metric with respect to all dimensions of internal coordinate space and external coordinate space. On-shell can be regarded as gradient direction. Quantum theory can be regarded as a geometric theory of distribution of gradient directions. Hence, gravitational theory and quantum theory obtain the same view of time and space and a unified description of evolution in affine connection representation of gauge fields. (iii) Chiral asymmetry, coupling constants, MNS mixing and CKM mixing can appear spontaneously in affine connection representation, while in $U(1) \times SU(2)\times SU(3)$ principal bundle connection representation they can just only be artificially set up. Some principles and postulates of the conventional theories that are based on principal bundle connection representation can be turned into theorems in affine connection representation, so they are not necessary to be regarded as principles or postulates anymore.