Hemispheric asymmetries differ considerably across species, but the neurophysiological base of this variation is unclear. It has been suggested that hemispheric asymmetries evolved to bypass interhemispheric conduction delay when performing time-critical tasks. This implies that large brains should be more asymmetric. We performed preregistered cross-species meta-regressions with brain mass and neuron number as predictors for limb preferences, a behavioral marker of hemispheric asymmetries, in mammals. Brain mass and neuron number showed positive associations with rightward limb preferences but negative associations with leftward limb preferences. No significant associations were found for ambilaterality. These results are only partly in line with the idea that conduction delay is the critical factor that drives the evolution of hemispheric asymmetries. They suggest that larger-brained species tend to shift towards more right-lateralized individuals. Therefore, the need for coordination of lateralized responses in social species needs to be considered in the context of the evolution of hemispheric asymmetries.