Born to Charles Jr. and Mary Matthews Just on August 14, 1883. His father and grandfather, Charles Sr., were builders. When Ernest was four years old, both his father and grandfather died (the former of alcoholism). Just's mother became the sole supporter of Just, his younger brother, and his younger sister. Mary Matthews Just, taught at an African-American school in Charleston to support her family. During the summer, she worked in the phosphate mines on James Island. Noticing that there was much vacant land near the island, Mary persuaded several black families to move there to farm. The town they founded, now incorporated in the West Ashley area of Charleston, was eventually named Maryville in her honor.