Interesting thought, considering the primary reason for rejecting the status in the first place is because they essentially  glorify a sense of  inaquality that many believe still exist in the United States. And while Lincoln may have rejected the concept of slavery, he also did not believe other races were the equal of white Europeans.  One quote of many from Lincoln:

"I will say in addition to this that there is a physical difference between the white and black races which I believe will forever forbid the two races from living together on terms of social and political equality. And inasmuch as they cannot so live, while they do remain together there must be a position of superior and inferior, and I as much as any other man am in favor of having the superior position assigned to the white race.”  

So if we are to reject status based on inaquality, we should at least be consistent in our argument as to rejecting all those who exemplify inaquality, not just a chosen few.