In this paper, we propose a vivid 3D pose driving (3PD) system containing 2D skeleton pose estimation (2D SPE) network, 3D association pose estimation (3D APE) network and 3D angle-matching (3D-AM) engine. The 2D SPE network consecutively predicts the joint coordinates of human 2D pose per frame for a video string. Then, the 3D APE network associates the successive 2D poses to predict the 3D skeletons of dynamic poses, which can be a causal or non-causal temporal convolutional network. The experimental results demonstrate that the integration of the 2D SPE network and the non-causal 3D APE network can generate accurate dynamic 3D skeletons. Then, the 3D-AM engine can apply those dynamic 3D skeletons to stably drive an on-line 3D avatar. By our 3PD system, the driven avatar can coherently follow user’s dynamic motions with natural agile activities for VR applications. The experimental results can demonstrate that the integration of 2D SPE and 3D APE networks can in usual attain the accurate 3D human-pose estimation without anyone auxiliary training appropriated to occluded joints. Besides, relative to the compared state-of-art networks, our integrated convolutional neural network (CNN) can have structural regularity and computational lightweight.