Barbour was born in Yazoo City, Mississippi, where he was raised as the youngest of three sons of Grace LeFlore (née Johnson) and Jeptha Fowlkes Barbour, Jr. Haley's father, a lawyer, died when Barbour was two years old. Barbour's father who was a Circuit Judge had an inmate to assist him when Judge Barbour became ill. Leon Turner, who was given a posthumous pardon by Barbour in the closing days of his administration, had helped to raise him.