Lord Desborough married Ethel Fane (27 June 1867 – 28 May 1952), daughter of the Hon. Julian Fane and granddaughter of John Fane, 11th Earl of Westmorland, in 1887. They had three sons and two daughters. His eldest son was the poet Julian Grenfell, who was killed in action in 1915. His second son, Gerald William Grenfell, was also killed about two months after his elder brother. His third son, Ivo George Grenfell, died in 1926 as the result of a car accident. As all his sons predeceased him, the barony became extinct. The family lived at Taplow Court, where he and his wife hosted gatherings of the elite and aristocratic group, the 'Souls', adjacent on the riverside to Cliveden, which is a slightly grander country estate, but which saw its social heyday immediately after, from 1920 to 1965. Lady Desborough was a well-known celebrity in her day. Margot Asquith, whose husband would later be politically aligned against Desborough, said of her, "She tells enough white lies to ice a wedding cake".