Saunders was born on a farm in Amherst County, Virginia, to Abram Warwick and Mary Gregory. Saunders mother died when he was five. Abram had served in the Confederate army under Stonewall Jackson but had struggled financially in the aftermath of the war. In 1891, his father moved the family to Montgomery County, Tennessee, where his father worked as a laborer and sharecropper on a plantation near Palmyra, Tennessee. Saunders worked the plantation as well throughout his early childhood. By age 11, he was also working in a sawmill and limestone kiln during the summer and at a general store during the holidays. He received only two years of formal education in Palmyra, before quitting school entirely at 14 to work full-time for the general store. Saunders worked a variety occupations over the next five years throughout the region, including shop assistant, night watchman and sawmill laborer. Recognizing the need for further education to advance, Saunders became a prolific reader and autodidact. At age 19, Saunders had moved to Clarksville, Tennessee, where he started working in the wholesale grocery business for John Hurst and Joseph Boillin, rapidly rising in the business.