This research aimed to establish an early-warning critical energy for coal instability based on the energy theory and acoustic emission characteristics of coal under triaxial compression. To obtain an early-warning critical strain energy indicating the increase in the risk of coal instability, conventional triaxial compression and acoustic emission (AE) tests were carried out on coal specimens taken from a 980-m-deep mine with initial confining pressures of 10, 15, 20, 25, 30 and 35 MPa. Stress-strain relations, AE features, and energy evolution characteristics during triaxial compression were analyzed. It was found that the energy evolution and AE event count changes across different loading stages. With increasing axial stress, most of the input energy stored in the coal specimens was in the form of elastic strain energy and the AE event count was close to zero, indicating that the coal grains reach a state of balance. After the elastic deformation stage, a portion of the input energy was consumed by inelastic deformation. Once the stress level exceeded the volumetric compressibility–dilatancy transition stress, the AE event entered a period of relative quiet, and the rate of energy dissipation abruptly accelerated, indicating that the coal grains achieved another state of balance before THE instability or failure. The balance of the rock grains is broken again (AE event count and the rate of energy dissipation both increased dramatically), coal achieved the peak strength and instability soon. The point at which the dissipated energy ratio α increased rapidly or the starting point of a quiet period, indicates an increase in the risk of coal instability. The corresponding elastic strain energy accumulated within the coal can be regarded as a precursor to instability or strainburst. Accordingly, a fitting formula is presented to predict the early-warning critical energy for brittle coal subject to different minimum principal stress. The analysis results in this paper can be helpful in the assessment of coal instability risk.