I grew up with 2 families that survived the camps. To this day, I despise swastikas, but see them constantly since my wife is India Indian and Hindu. 
Likewise, a number of relatives fought for the union and 1 of my relatives was part of the underground railroad and disappeared (very likely suffering far far more than what run away slaves did before he died; Confederates HATED whites that helped while slaves were considered property/animals ). 

Any statue from say late 20s backwards, should be saved. Anything newer, let them go. Most, if not all, of the newer ones was not memorials, but about trying to sway ppl. 
Likewise, renaming buildings, esp those from the 30s onwards, is fine.
The state flags need to be redesigned since many of the southern ones include elements of Confederacy. 
But DESTROYING historical statues/sites, etc, well, that is just plain wrong.