You are wrong.

https://en.wikipedia.org/wiki/Slavery_in_the_United_States

See specifically the segments on Native Americans, Black slaveholders, Intertribal slaveholders, Barbary pirates, and asian slavery.  

That ignored the virtual slavery of indentured servitude and servitude imposed as the condition of a criminal conviction. Everyone even slaves had rights (of one sort or another) and depending upon the area of the country those rights varied significantly (see NEW ORLEANS)