Following a severe disaster, a quick and reliable evaluation of the structural damage state is one of the crucial steps to making decisions and disaster management. Based on a realistic estimate of types and levels of damages, decisions related to the structural performances, repair possibility, or in severe cases, a replacement could be made. In available technical literature, reporting the classification of damaged buildings is limited and based on numerical studies, laboratory tests, and inspector’s judgment. Presented results in current research are from a comprehensive and meticulous investigation of more than 81 damaged steel and RC buildings after the Sarpol-e Zahab (Iran) earthquake. Considering this valuable collected information, a novel qualitative-quantitative approach is introduced to classify damaged steel and RC buildings into 5 damage states, both for structural and non-structural components. For this purpose, weights are attributed to each type of damage, according to the severity of the damage observed in each component and its influence on the damage to the entire building. Then, the damage index of each structure is obtained using the proposed relation, and results are compared with field observation for verification. Finally, empirical vulnerability curves of investigated buildings are developed based on the proposed damage index and spectral acceleration at fundamental periods of structures as an intensity measure.