Avery Brooks was born in Evansville, Indiana, the son of Eva Lydia ("née" Crawford), a choral conductor and music instructor, and Samuel Brooks, a union official and tool and die worker. His maternal grandfather, Samuel Travis Crawford, was also a singer who graduated from Tougaloo College in 1901. When Avery was eight years old his family moved to Gary, Indiana, after his father had been laid off from International Harvester. Brooks has said: "I was born in Evansville... but it was Gary, Indiana that made me."