I guess by your reasoning the only place any human could legitimately be "native" to would be Africa, the cradle of humanity.  Therefore making the word quite useless in the context of referring to any indigenous populations.