Arnold married Mary Penrose, daughter of the Rev. John Penrose of Penryn, Cornwall. They had five daughters and five sons, including the poet Matthew Arnold, the literary scholar Tom, the author William Delafield Arnold and Edward Penrose Arnold, an inspector of schools. One daughter died in infancy. The eldest daughter, Jane Martha, married William Edward Forster. Both enjoyed mountaineering; they climbed Mont Blanc in 1859 and in 1860 Jane was one of the first women to stand on the summit of Monte Rosa, which had not been climbed by a woman until 1857. When William Delafield Arnold died in 1859 leaving four orphans, the Forsters adopted them as their own, adding their name to the children's surname. One of them was Hugh Oakeley Arnold-Forster, a Liberal Unionist MP, who eventually became a member of Balfour's cabinet. Another was Florence Vere O'Brien, a diarist, philanthropist and craftswoman who lived in Ireland. Frances Bunsen Trevenen Whateley Arnold, the youngest daughter, never married and died at Fox How in 1923.