Millimeter-scale magnesian refractory granules were found to be a unique magnesian-expansive component in steel slag. To systematically study the effects of these granular magnesian-expansive components on the volume stability of cement-based materials containing steel slag, an investigation of their existing forms and influence on the volume stability was conducted in this paper. The various-sizing waste–magnesium–chromate-based refractory brick (Mg-Cr brick) granules and different (FeO + MnO)/MgO ratios’ synthetic MgO·xFeO·yMnO ternary solid solutions granules were adopted to simulate magnesian-expansive granules by partially replacing manufactured sand in mortar. The 100 °C–3 h boiling and 213 °C–2 MPa–3 h autoclaving treatments were adopted as volume stability testing methods. The results indicated that whether Mg-Cr brick or MgO·xFeO·yMnO solid solution, the concentration of expansive stress and the anisotropy expansion came with the granular size rising weakening the volume stability of cement-based materials which contained magnesian-expansive granules, significantly. Meanwhile, this phenomenon resulted in the ineffectiveness of the single linear expansion rate when assessing the qualification of volume stability. Furthermore, it also changed the mortars’ failure mode from “muddy damage” to “break into blocks”. Especially, there is no volume stability issue when the MgO·xFeO·yMnO satisfied (FeO + MnO)/MgO ≥ 1.00. Considering the significant effect of the granular magnesian-expansive components on the volume stability of cement-based materials containing steel slag, it is imperative to enhance the detection of both MgO content and mineral existing forms in steel slag in practical applications. For recommendation, the threshold value of conducting autoclaved volume stability testing on steel slag should be set at MgO ≥ 3%. Furthermore, the qualification cannot be judged by the single linear expansion rate; the specimens’ appearance integrity and strength loss should also be noted.