The study of fracture and instability mechanism of the main roof in steeply dipping coal seams (SDCS) workface is crucial to the proper choice of support type and control of stability of surrounding rock, as well as the safe and effective mining in the coal seam. Based on the established SDCS main roof model, this study derived the stress distribution in the main roof under linear load, analyzed the dip angle effect associated with the evolving stress of SDCS workface, and elaborated the sequential characteristics of the ground pressure mechanism. Besides, an inclined unstable structure model of the main roof based on deformation, fracturing, and rotating of the main roof in the SDCS workface was also proposed here, which explained the impacts of overburden rock's key parameters on sliding and rotatory instability of rock mass. In light of the analysis of the movement rule of overburden rock and loading condition of support, it is found that, when the roof and floor in the workface are stable, the critical support resistance with the absence of sliding and rotating increases as the dip angle of coal seam increases, the roof is in the state of discontinuous movement due to its self-weight and overburden pressure. Support is affected by the discontinuous movement and moved along with the roof. The results of this study can be of theoretical reference to the control of SDCS.