There were "hardly any" blacks in Oregon because Oregon was socially and, importantly, legally hostile to them. The legal hostilities weren't expunged until late in the last century.
Look up "A Peculiar Paradise" online, a good history of Oregon racial laws and relations.