After a relationship in the mid-1860s with the novelist Annie Thomas, Gilbert married Lucy Agnes Turner (1847–1936), whom he called "Kitty", in 1867; she was 11 years his junior. He wrote many affectionate letters to her over the years. Gilbert and Lucy were socially active both in London and later at Grim's Dyke, often holding dinner parties and being invited to others' homes for dinner, in contrast to the picture painted by fictionalisations such as the film "Topsy-Turvy". The Gilberts had no children, but they had many pets, including some exotic ones.