As an important link in the complex system engineering project of open pit mining, the quality of the boundary determines the performance of the project to a large extent. However, changes in economic indicators may raise doubts about the optimality of mining boundaries. In this article, a coal price time series forecasting model that considers some amount of redundancy is proposed, which combines an improved sparrow search algorithm (ISSA) and a least squares support vector regression machine regression (LSSVR) algorithm. The optimal values of the penalty factor and kernel function parameter of the LSSVR model are selected by ISSA, which improves the prediction accuracy and generalization performance of the forecasting model. A multistep decision optimization method under fluctuating coal price conditions is proposed, and the model prediction results are applied to the boundary optimization design process. Using the widely applied block model as the basis, a set of optimal production nested pits is obtained, allowing the realm design results to fit the coal price fluctuation trend and further enhance enterprise efficiency. The applicability and effectiveness of this method were verified by taking an ideal two-dimensional model and an inclined coal seam open-pit coal mine in Xinjiang as an example.