You don't understand the fact that the statues glorify the leaders of a war fought to preserve slavery?  Also, are you unaware of the fact that many of the statues were erected during the Jim Crow era?  Here is a neutral source of this information:  http://www.townandcountrymag.com/society/politics/a12015570/confederate-monuments/.  People who support the statues are supporting white supremacy, because that is what the statues stand for. 
Those who want to spout white supremacist speech are allowed to do so, but those who oppose that speech are also allowed to voice their opposition.
Those who support white supremacy are the ones who "seek to fracture this nation."