Kelley married the former Malinda Hufford in 1832, shortly before they moved to Columbus. The Hufford family was of Swiss extraction, having originally settled in Virginia before coming to Ohio in 1831. Kelley is listed as head of household in the 1840, 1850 and 1870 U.S. censuses as residing in Columbus, and he and his family are enumerated three times in 1860 :in Columbus, the 6th ward of Cincinnati and in Hopkinsville, Christian County, Kentucky. Census records, burial records and city directories make it possible to state that Kelley had two sons, William George (1843–1867) and James H (1847–1876). He had at least one daughter, perhaps two, but records are contradictory. Martha Jane Kelley (1835–1836) is buried in Green Lawn beside her father and brothers. A Maria Kelley, aged 3, is part of the 1850 census listing, but appears in no other census listings, and is not mentioned in probate records when Kelley died.