Sullivan married Charlotte Louisa Lacy (1841–1885) in 1862, and they had eight children: Amy Sophie (1863–1947), Florence Louise (1865–1891), Edith Mary (1866–1877), Herbert Thomas "Bertie" (1868–1928), Maud Helen "Cissie" (1870–1940), Frederic Richard ("Dickie") (1872–1937), George Arthur (1874–1919), and William Lacy (1877–1902). At Sullivan's death, Charlotte was pregnant and had seven children under the age of 14. Edith, however, died shortly after her father. After Fred Sullivan's death, Arthur visited the family often and became guardian to all of the children. He was closely involved in their lives, hosting them on European summer holidays. Even before Fred's death, Amy lived with Arthur and her grandmother in London for a year after an 1875 illness. In 1881, Charlotte married Captain Benjamin Hutchinson, a man 13 years her junior.