Allen was born in Scottsville, Virginia on July 14, 1919. He received no formal schooling. He began working as a waiter at a pool club. Allen met his wife Helene in 1942. They had a son, Charles Allen. Helene wanted to vote for Barack Obama in 2008, but she died the day before the election, on November 3, 2008. 