Backgroundwe assessed the performance of the optimization algorithms by comparing volumetric modulated arc therapy generated by a progressive resolution optimized (VMATPRO) and photon optimizer (VMATPO) in terms of plan quality, MU reduction, sparing of the spinal cord (or cauda equina), and plan complexity.MethodsFifty-seven patients who received spine stereotactic ablative radiotherapy (SABR) with tumors located in the cervical, thoracic, and lumbar spine were retrospectively selected. For each patient, VMATPROand VMATPOwith two full arcs were generated with using the PRO and PO algorithms. For dosimetric evaluation, the dose-volumetric (DV) parameters of the planning target volume (PTV), organs at risk (OARs), the corresponding planning organs at risk (PRV), and 1.5-cm ring structure surrounding the PTV (Ring1.5 cm) were calculated for all VMAT plans. The total number of monitor units (MUs) and the modulation complexity score for the VMAT (MCSv) were compared. To investigate the correlations of OAR sparing to plan complexity, Pearson’s and Spearman’s correlation tests were conducted between the two algorithms (PO – PRO, denoted as Δ) in the DV parameters for normal tissues, total MUs, and MCSv.ResultsFor the PTVs, Target conformity and dose homogeneity in the PTVs of VMATPROwere better than those of VMATPOwith statistical significance. For the spinal cords (or cauda equine) and the corresponding PRVs, all of the DV parameters for VMATPROwere markedly lower than those for VMATPO, with statistical significance (allp< 0.0001). Among them, the difference in the maximum dose to the spinal cord between VMATPROand VMATPOwas remarkable (9.04 Gy vs. 11.08 Gy withp< 0.0001). For Ring1.5 cm, no significant difference in V115%for VMATPROand VMATPOwas observed.ConclusionsThe use of VMATPROresulted in improved coverage and uniformity of dose to the PTV, as well as OARs sparing, compared with that of VMATPOfor cervical, thoracic, and lumbar spine SABR. Better dosimetric plan quality generated by the PRO algorithm was observed to result in higher total MUs and plan complexity. Therefore, careful evaluation of its deliverability should be performed with caution during the routine use of the PRO algorithm.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12885-023-10925-z.