Elam was born near Hope in Hempstead County in southwestern Arkansas, to William Jefferson Elam, a teacher, and his wife, the former Cynthia Wheaton, both from Virginia. The Elam family moved in 1823 to Ayish Bayou near San Augustine in East Texas, where another son, Charles Wheaton Elam, was born. In 1826, the Elams relocated to Natchitoches, Louisiana, where Mary Jane Elam, was born. By 1833, the family moved to Fort Jessup, Louisiana, the westernmost outpost of the United States at that time. William Elam tutored the officers' sons. Another son, John Waddill Elam, was born at Fort Jesup in 1833. There was also a daughter, Henrietta Elam.