There is a difference between memorializing those who died in a war, vs memorializing the leaders that started and led that war.  The Vietnam memorial pays homage to the Americans who died, not the politicians and generals who started and fed that war.  There is a difference and to be honest, it isn't really subtle.  Statues of Lee and Davis are not there to memorialize the fallen soldiers of the confederacy.  They are there to glorify the confederacy.  It would be akin to having statues of Hitler and Rommel in Germany.  Would you still suggest they shouldn't be taken down because of "history"?