John Frederick's only son, George, the fourth duke, died in 1815 at 21, and Knole was then left by the 3rd Duke's widow in 1825 to their daughter Mary, Countess of Plymouth. She died childless in 1864, leaving it to her sister Countess De La Warr and her heirs male. It ultimately passed to the latter's fourth son, Mortimer Sackville-West, 1st Baron Sackville and thence to his successors. However, lord Sackville's resources were insufficient to maintain the house and its possessions. He began selling a number of the heirlooms to enable him to keep the estate going.