Albert Benjamin Chandler was born in the farming community of Corydon, Kentucky, in 1898. He was the eldest child of Joseph Sweet and Callie (Saunders) Chandler. Chandler's father allegedly rescued his mother from an orphanage and married her when she was 15, but no record of their marriage has ever been found. In 1899, Chandler's brother Robert was born. Two years later, their mother, still in her teens and unable to cope with raising two young children, abandoned the family. She fled the state and left her sons with their father. In his autobiography, Chandler said that his mother's leaving them was his earliest memory. Years later, he sought his mother and found her living in Jacksonville, Florida. She had married again and he had three half-siblings. His full brother, Robert Chandler, died when he fell from a cherry tree when he was 13 years old.