Smith was born in Grand Junction, Colorado on November 19, 1909. At the age of ten, his mother died, and at age thirteen, his father died, leaving young Elmo an orphan. He was sent to live with his uncle on a ranch near Wilder, Idaho. He supported himself financially as he pursued an education at the College of Idaho in Caldwell. Smith received his B.A. in History in 1932, and moved to nearby Ontario, Oregon, just across the Snake River from Idaho.