About 1828, James and his father moved to Claremont, New Hampshire, where he received a limited education. In 1838, Nesmith moved to Ohio, followed by Iowa in 1842 where he waited to immigrate to Oregon Country.
James Nesmith was born in what is now the Canadian province of New Brunswick (which was a British colony at the time) while his parents were on a visit from their home in Washington County, Maine, on July 23, 1820. Of Scottish and Irish heritage, his father was William Morrison Nesmith and his mother the former Harriet Miller.