In polygynous populations, where only a few males leave offspring, there is a much smaller reproducing population than if all viable males were considered.
In societies with multiple mates, where only a few males contribute to the next generation, the actual population size for reproduction is much smaller than if all capable males were taken into account.