Handy was born in Florence, Alabama, the son of Elizabeth Brewer and Charles Barnard Handy. His father was the pastor of a small church in Guntersville, a small town in northeast central Alabama. Handy wrote in his 1941 autobiography, "Father of the Blues", that he was born in a log cabin built by his grandfather William Wise Handy, who became an African Methodist Episcopal minister after the Emancipation Proclamation. The log cabin of Handy's birth has been preserved near downtown Florence.