>Grossly offensive statues that only represent clear bigotry should be removed - I doubt many would object. But the debate is not about those is it?
Yes it absolutely is. Robert E. Lee (or for that matter Cecil Rhodes) is no more deserving of a statue than Hitler or Stalin - and I didn't hear you complaining when Europe and Russia tore down every statue of those two they could find ! 

The only difference, the ONLY difference between them is that Lee and Rhodes lived in a time when being a genocidal white supremacist was still rather popular with many white people.