Epistasis affects genome evolution together with our ability to predict individual mutation effects. The mechanistic basis of epistasis remains, however, largely unknown. To quantify and better understand interactions between fitness-affecting mutations, we focus on a 11 amino-acid α-helix of the protein β-lactamase TEM-1, and build a comprehensive library of more than 15,000 double mutants. Analysis of the growth rates of these mutants shows pervasive epistasis, which can be largely explained by a non-linear two-state model, where inactivating, destabilizing, neutral, or stabilizing mutations additively contribute to the phenotype. Hence, most epistatic interactions can be predicted by a non-linear model informed by single-point mutational measurements only. Deviations from the two-state model are consistently found for few pairs of residues, in particular when they are in contact. This result, as well as single-point mutation parameters, can be quantitatively found back through direct-coupling-analysis-based statistical models inferred from homologous sequence data. Our results thus shed light on the existence and the origins of the multiple determinants of the epistatic landscape, even at the level of small structural components of a protein, and suggest that the corresponding constraints shape the entire β-lactamase family.