Key messageReciprocal recurrent selection sometimes increases genetic gain per unit cost in clonal diploids with heterosis due to dominance, but it typically does not benefit autopolyploids.AbstractBreeding can change the dominance as well as additive genetic value of populations, thus utilizing heterosis. A common hybrid breeding strategy is reciprocal recurrent selection (RRS), in which parents of hybrids are typically recycled within pools based on general combining ability. However, the relative performances of RRS and other breeding strategies have not been thoroughly compared. RRS can have relatively increased costs and longer cycle lengths, but these are sometimes outweighed by its ability to harness heterosis due to dominance. Here, we used stochastic simulation to compare genetic gain per unit cost of RRS, terminal crossing, recurrent selection on breeding value, and recurrent selection on cross performance considering different amounts of population heterosis due to dominance, relative cycle lengths, time horizons, estimation methods, selection intensities, and ploidy levels. In diploids with phenotypic selection at high intensity, whether RRS was the optimal breeding strategy depended on the initial population heterosis. However, in diploids with rapid-cycling genomic selection at high intensity, RRS was the optimal breeding strategy after 50 years over almost all amounts of initial population heterosis under the study assumptions. Diploid RRS required more population heterosis to outperform other strategies as its relative cycle length increased and as selection intensity and time horizon decreased. The optimal strategy depended on selection intensity, a proxy for inbreeding rate. Use of diploid fully inbred parents vs. outbred parents with RRS typically did not affect genetic gain. In autopolyploids, RRS typically did not outperform one-pool strategies regardless of the initial population heterosis.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00122-023-04377-z.