Hello folks, a cool quote from a textbook:

"The development of cooperative hunting of large mammals by our ancestors led to the formation of cohesive all male hunting groups" (from Social Behavior in Animals).

Anyways before that, way, way, way back, those earlier mammals were likely hunters, namely insectivores, and, because of their very tiny size, it is likely that the females had to be wrapped among her young to keep them warm, both by providing milk and sharing her body heat by direct contact. The smaller your young, the more rapidly they cool, the more often they need feeding. In the existent small insectivores still alive to this day, the males have much larger territories than females, again getting back to greater demands on spatial perception and geometry. And the act of males bringing adult females food to aid in steps of embryo/fetal/young development? That predates considerably the last common ancestor of mammals, being present in birds and, it is thought, dinosaurs!