Charts are powerful tools for visualizing and comparing data. With the increase in the presence of various chart types in scientific documents in electronic media, the development of automatic chart classification system is becoming an important task. Existing studies on chart classification fail to address the presence of noise in charts, and confusing chart class pairs. Motivated by the above observations, in this paper, we propose an attention and triplet loss based deep CNN framework to address the above issues. From various experimental results over four datasets, it is evident that the proposed framework can effectively handle noises in the chart and confusing chart samples, and outperforms its counterparts.