Genetic drift is the name given to this loss of variation. It can affect evolution in surprisingly big ways, but "only when populations are quite small". What 'small' means would depend on the organism. 50 individuals is small, 5000 is not, 500 is maybe. The basic mechanism of drift is that genetic variety is reduced by chance, making the individuals more similar to each other, and hence more vulnerable.