No, both sides didn't go looking for a fight.  There were those there from both sides looking for a fight but not all against the hateful racist ideology went there looking for a fight, but rather just to support that this type of ideology has no place whatsoever in America.  White Supremacist ideology bears striking resemblance at it's fundamental core to Islamic extremism....brainwashed mentality and hatefully violent thoughts directed at another group of people.  While it's true they have a "right" of free speech, hate speech IS  profanity and  regulated as intolerable in public space.  White supremacist hate ideology was  in public space and that is intolerable and not a "right".  
"The Civil War was about keeping fellow humans in bondage."  The core of the "make America Great again" Trump propaganda are the Kochs and Mercers etc,  whose agenda is to keep the wealthy on top and the rest of the population slaving to support the elite 1%.   Nothing's changed except the characters.