Jesse N. Smith was born the youngest of three sons to Silas Smith (1779–1839) and his second wife Mary Aikens (1797–1877) in Stockholm, New York. Both of Smith's grandfathers, Asael Smith (1744–1830) and Nathaniel Aikens (1757–1836), served in the American Revolutionary War. According to Smith, his grandfather Aikens served under General George Washington. Smith's father, Silas, married his first wife, Ruth Stevens, in 1806. Together they had seven children: Charles, Charity, Curtis Stevens, Samuel, Stephen, Susan and Asahel. After the death of his first wife, Silas courted Mary Aikens while she was teaching school in Stockholm and they married in 1828. The oldest child of this union was Silas Sanford Smith (1830–1910) followed by John Aikens Smith (1832–1838). Unfortunately John died due to exposure and unfit conditions from mob persecutions.