Material recognition using optical sensors is a key enabler technology in the field of automation. Nowadays, in the age of deeplearning, the challenge shifted from (manual) feature engineering to collecting big data. State of the art recognition approachesare based on deep neural networks employing huge databases. But still, it is difficult to transfer these latest recognition results into the wild – various lighting conditions, a changing imagequality, or different and new material classes are challenging complications. Evaluating a larger electromagnetic spectrum isone way to master these challenges. In this study, the infrared (IR) emissivity as a material specific property is investigated regarding its suitability for increasing thematerial classification reliability. Predictions of a deep learning model are combined with engineered features from IR data.This approach increases the overall accuracy and helps to differentiate between materials that visually appear similar. Thesolution is verified using real data from the field of automatized disinfection processes.