He married Martha E McCarty in 1846 and they had two children, William and John Lea. Smith later married Lavinia W. Henderson, daughter of Hon. Luke Lea, U. S. Representative for Tennessee, in 1855.
He was the younger brother of Colonel M. Whit Smith (b. Oct. 7, 1814), prominent lawyer, politician and Confederate officer from Florida. Smith received a limited education, taught school, and studied law. He was admitted to the bar in 1845 and commenced practice in Cleveland, Tennessee.