Smith, known to his friends as "Baldy", was born at St. Albans, Vermont, the son of Ashbel and Sarah Butler Smith, and a cousin of J. Gregory Smith (governor of Vermont, 1863–1865). He was educated locally in Vermont until he attended the United States Military Academy at West Point in 1841, and graduated four years later standing fourth of 41 cadets. Smith was appointed a brevet second lieutenant on July 1, 1845, and was assigned to the Topographical Engineers Corps. He was promoted to second lieutenant on July 14, 1849, and promoted to first lieutenant on March 3, 1853.