Biologist: Some speculate that the unusually high frequency of small goats found in island populations is a response to evolutionary pressure to increase the number of goats so as to ensure a diverse gene pool  However, only the reproductive success of a trait influences its frequency in a population
So, the only kind of evolutionary pressure that can reduce the average size of the members of a goat population is that resulting from small goats achieving greater reproductive than their larger cousins
The evolutionary pressure to ensure a diverse gene pool does not have the effect of increasing the frequency of a gene for small size.