Michael Abiona, 34, was greeted with posters reading "Local Houses 4 Local People" at Glenluce Drive, east Belfast, on Tuesday.
Mr Abiona who has been living in Northern Ireland since 2010 said four women and a man were outside the house.
They asked how he had managed to get the house and whether he was disabled.
Now, he said, he will not move into the Housing Executive property at Garnerville because the mother of his son is frightened and will not allow their child to visit the house.
Mr Abiona said he had reported what happened to the police.
Mr Abiona is active in race relations in Northern Ireland and  serves in key roles in the Northern Ireland Council for Ethnic Minorities and  the Nigerian Association.
Anti-racist rallies have been held across Northern Ireland in recent weeks following a spate of racist attacks.
It also follows a public apology by First Minister Peter Robinson after he said he would not trust Muslims involved in violence or devotees of Sharia law, but would trust Muslims "to go to the shops" for him.
He claimed he was misinterpreted, and apologised to Islamic leaders in Belfast.
He had been speaking in support of Belfast pastor James McConnell who described Islam as "heathen" and "satanic". He also apologised for his comments.
A man from Nigeria has said he is afraid to move  into his new Belfast home after he arrived to find five people with banners on the doorstep.