Surface energy is fundamental in controlling surface properties and surface-driven processes like heterogeneous catalysis, as adsorption energy is. It is thus crucial to establish an effective scheme to determine surface energy and its relation with adsorption energy. Herein, we propose a model to quantify the effects of materials’ intrinsic characteristics on the material-dependent property and anisotropy of surface energy, based on the period number and group number of bulk atoms, and the valence-electron number, electronegativity and coordination of surface atoms. Our scheme holds for elemental crystals in both solid and liquid phases, body-centered-tetragonal intermetallics, fluorite-structure intermetallics, face-centered-cubic intermetallics, Mg-based surface alloys and semiconductor compounds, which further identifies a quantitative relation between surface energy and adsorption energy and rationalizes the material-dependent error of first-principle methods in calculating the two quantities. This model is predictive with easily accessible parameters and thus allows the rapid screening of materials for targeted properties.