little benjamin always missed his father when he went away and worried about him because he had heard that where he worked the white men were very cruel , and it was a dangerous place for a black man to live .