Fisher married Frances Katharine Josepha Broughton, the daughter of the Rev. Thomas Delves Broughton and Frances Corkran, on 4 April 1866 while stationed at Portsmouth. Kitty's two brothers were both naval officers. According to a cousin, she believed that Jack would rise "to the top of the tree." They remained married until her death in July 1918. They had a son, Cecil Vavasseur, 2nd Baron Fisher (1868–1955), and three daughters, Beatrix Alice (1867–1930), Dorothy Sybil (1873–1962), and Pamela Mary (1876–1949), who all married naval officers who all went on to become admirals. Beatrix Alice married Reginald Rundell Neeld in 1896, Pamela Mary married Henry Blackett in 1906, and in 1908 Dorothy Sybil married Eric Fullerton.