Harold grew up on Horn Lake Road in the West Junction neighborhood of South Memphis. He is the eighth of fifteen children born to Newton Jackson Ford (1914–1986) and Vera (Davis) Ford (1915–1994), prominent members of the African-American community. His mother was a homemaker and his father was an undertaker and businessman, who opened N.J. Ford Funeral Home (later changed to N.J. Ford And Sons Funeral Home) in 1932. His grandfather Lewie Ford (1889-1931) started the family funeral business and became allied with E.H. Crump, an influential white politician in Memphis and the state in the early 20th century.