Materials with high stiffness and good vibration damping properties are of great industrial interest. In this paper, a topology optimization algorithm based on the BESO method is applied to design viscoelastic composite material by adjusting its 3D microstructures. The viscoelastic composite material is assumed to be composed of a non-viscoelastic material with high stiffness and a viscoelastic material with good vibration damping. The 3D microstructures of the composite are uniformly represented by corresponding periodic unit cells (PUCs). The effective properties of the 3D PUC are extracted by the homogenization theory. The optimized properties of the composites and the optimal microscopic layout of the two materials phases under the conditions of maximum stiffness and maximum damping are given by several numerical examples.