Smith was twice married, and left by his first marriage (to Anne Read) two daughters Caroline and Emily; by his second marriage (to a daughter Mary of John Christie of Hackney) an only son, Herman (died 23 July 1897, aged 77). Miranda and Octavia Hill were his granddaughters, among the five daughters of Caroline (c. 1809–1902) who married James Hill, merchant in 1835, and as Caroline Southwood Hill was known as a writer and educationalist. His other daughter was Emily (born 1810).