Kelley was of Anglo-Irish heritage, his great-grandfather John Kelley leaving Belfast, Ireland to settle in New Castle, Delaware. The architect was named for his paternal grandfather, Nathan Kelley (1760–1845), who left Delaware to head west, first to south western Pennsylvania, where he married Hannah Miller, and then to Warren County in what was then the Northwest Territory. The elder Nathan was a prosperous farmer who served the area as justice of the peace, appeals court judge and state representative. Nathan B. Kelley was the third of 8 children born to James Miller Kelley and Rebecca Ludlum Kelley. The Ludlum line can be traced back to Derbyshire, England. The Kelley family had a demonstrated habit of sharing names among generations and branches of the family, so that Nathan had both uncles and brothers with the same names. He had a nephew, Nathan Kelley Hufford, named in his honor. His elder brothers Thomas Jefferson Kelley and George Washington Kelley demonstrate the patriotic zeal felt in the still young nation.