Backpropagation is the most popular and common algorithm for training of traditional deep neural networks. Herewe propose temporal version of backpropagation to directly train spiking neural networks with deep structure andsingle spikebased temporal coding scheme (DS4NN). We consider a convolutional spiking neural networkconsisting of simple non-leaky integrate-and-fire (IF) neurons, and a temporal coding known as timeto-first-spikecoding. These together lead to lower computational cost and higher inference speed. We use surrogate gradient atfiring times to solve the non-differentiability of spike times with respect to membrane potential of spiking neurons,and to prevent the emergence of dead neurons in deep layers, we propose a relative encoding scheme fordetermining desired firing times. Evaluations on two classification tasks of MNIST and FashionMNIST datasetsconfirm the capability of DS4NN on deep SNNs. It achieves the accuracy of 99.3% and 91.6% on respectivelyMNIST and FashionMNIST datasets with the mean required number of 1126 and 1863 spikes in the whole network.This shows that the proposed approach can make fast decisions with low cost and high accuracy.