His father is depressed and troubled from working hard for "the white man" for so many years, yet having nothing to show for it.
The speaker's description conveys the emotional toll of their father's labor, highlighting his sense of disillusionment and desperation despite his dedication to working for others.