Encrypted traffic classification plays a critical role in network management, providing appropriate Quality-of-Service and Network Intrusion Detection. Conventional port-based and deep packet inspection (DPI) approaches cannot classify encrypted traffic effectively. Methods based on machine learning can classify encrypted traffic by extracting statistical features of the flow. However, they require manual extraction of features. Recent studies show that the approaches based on deep learning are compelling for the task. They can automatically learn raw traffic features without manual feature extraction. However, these studies still take the payload of encrypted traffic as the model input, which may cause privacy risks. Besides, a massive encrypted payload causes great storage pressure on traffic classification. In this paper, we propose a reliable encrypted traffic classification framework by only using the flow header called Only Header, which avoids privacy risks and achieves lightweight storage. Firstly, we introduce a twice segmentation mechanism to dilute the interference traffic and increase the weight of effective traffic. Then we use capsule neural networks (CapsNet) to learn spatial and byte features of the flow header. The Only Header's effectiveness is compared with other methods using two public datasets, including ISCX VPN-nonVPN and ISCX Tor-nonTor datasets.