Notice that in Genesis 3:15-16 the woman, is spoken of in both the spiritual and natural context meaning we cannot separate the two. 
The same is shown in Ephesians 5:22-24.

We should remember that Adam and Eve were the first members of God's church on earth. God already said the man is to rule the woman (both as wife and as church). Thus within a church setting, God prefers men to rule the flock.

In 1 Timothy 3:4, 5, when talking of the Bishops, Paul said they were to be rulers in their homes. In Genesis we already saw that the position of ruling was given to the man, not the woman. If he ruled his home well, he could be trusted to rule the church.

So when we talk of Elders and Pastors, we are talking about rulers. We already saw which gender he chose to rule the church. In both the home and the church setting, things are the same.