On 6 September 1864 Woolner married Alice Gertrude Waugh. He had initially been in love with her sister, Fanny, and had previously proposed to her, but she turned him down. Fanny married Woolner's Pre-Raphaelite colleague William Holman Hunt the following year, but died in childbirth a year later. In 1874, while in Italy, Hunt married their third sister Edith, an act which Woolner considered immoral and which was defined as incest under British laws at the time. He never spoke to Hunt again.