When highwall mining technology is applied to recover large amounts of residual coal left under the highwall of a big open-pit mine, reasonable coal pillar width is the premise for maintaining the stability of web pillars. By adopting the numerical simulation method, the characteristics of the abutment stress distributions in the web pillars under different slope angles and mining depths are studied, and the function of the stress distribution in the web pillar is established. The relationship between the abutment stress and the ultimate strength of the web pillar under different widths is also analyzed and used in combination with the failure characteristics of the pillar yield zone to explore the instability mechanism of the web pillar. The retaining widths of the web pillars are determined. Based on the modeling results, a mechanical bearing model of the web pillar is established, a cusp catastrophe model of pillar-overburden is constructed, and the formula for the web pillar instability criterion is obtained. By analyzing and calculating the ultimate strength of the web pillar, the formula for calculating the yield zone width at both sides of the pillar is achieved. Using the instability criterion of web pillars in highwall mining, a reasonable pillar width can be deduced theoretically, which provides significant guidance on the application of highwall mining technology.