Simon took his "not racist, not racist" quote completely out of context.   

Gavin is not a racist.  He has successfully sued people for claiming he is a Nazi, white supremacist and other things.   Ironically Gavin was talking about shady journalists who write smear  campaigns against him.

Gavin dropped out of the "Unite the Right" rally after seeing that it was not upholding conservative values.  And that people he does not and can not align with were running the show.  Which many other folks did.  

He condemns racism, yet Simon is either ignorant to that fact or doesn't care.