"...and if you find yourself surrounded by people with swastika flags shouting 'Blood and soil' and they agree with your world viewpoints, that's on you. "

That is why he refused to go to the "Unite the Right" rally.  Once he learned that there were white supremacists and Nazis leading the rally - 2 groups he openly mocks and denounces - he condemned it.  

Constitutional conservatives do not get along with the Alt-Right.  And Gavin also openly condemns the Alt-Right.   They do not like him, because of his viewpoints and beliefs.  And the fact that he has bi-racial children.