As an important link in the complex system engineering project of open-pit mining, the quality of boundary determine the performance of the project to a large extent. However, the traditional design method cannot effectively measure the impact of uncertainties on the realm optimisation process. In this article, a coal price time series forecasting model that considers the amount of redundancy is proposed, which combines an improved sparrow search algorithm (ISSA) and a least squares support vector regression machine regression algorithm (LSSVR). The optimal values of the penalty factor and kernel function parameter of the LSSVR model are selected by ISSA, which improves the prediction accuracy and generalisation performance of the forecasting model. A multi-step decision optimisation method under fluctuating coal price conditions is proposed, and the model prediction results are applied to the boundary optimisation design process. Using the widely applied block model as the basis, a set of optimal production nested pits is obtained and only obtained, allowing the realm design results to fit the coal price fluctuation trend and further enhance the enterprise efficiency. The applicability and effectiveness of the proposed method is verified using an ideal two-dimensional inclined coal seamopencast mine model as an example.