It is much more violent to encourage and incite rape and death threats and the deportation of lawful residents of the United States then it is to set a streetlamp on fire. 

Yiannopoulos uses his star appeal amongst the fascists, racists, and sexists to threaten and endanger the lives of people of color and women. He encourages his rabid fans to shut down any and all of his critics with abuse, both physical and psychological. That is an attack on free speech.

Remember that it was a supporter of Yiannopoulos that shot a man in Seattle only a handful of days ago at another talk by the man.