If you go over to Duck U, there is a place called Deady Hall.  Sometimes, that causes people to ask "Who was Deady".  

You then discover a founder of Duck U who wrote a Constitution forbidding Negroes to live in the State and forbidding slavery.  You also discover that later, as a Judge, he defended Chinese when they were being discriminated against.  

With no name on the building, as a passerby, you would never discover any such thing.

There are statues to Caligula, one of the most vile Romans in history.  "Who was Caligula?" is less likely to happen without such a statue.  

Your argument in favor of monument removal is specious.