Brown was born on 13 October 1926 in Hattiesburg, Mississippi. He was one of six children born to Julia Lindsey Brown, a schoolteacher, and John Brown, a grocery warehouse worker. He had four brothers, Marvin, William, Fletcher, and Lura, as well as an older sister known as Johnny. Brown's ancestry was African American, Chickasaw, and Choctaw. The family lived in a house without central heating or indoor plumbing so they relied on a fireplace for warmth. As a child, Jesse's brother William fell into this fireplace and was severely burned.