It is indeed troubling that Clinton could be driven out by voters who don't necessarily deplore his conduct.
The voters don't necessarily oppose Clinton's conduct.
Clinton was driven out by a group of angry voters who hated his conduct.