George Shirley, Esq. (created a baronet, on the first institution of that order, by James I. in 1611), Lovett's grandson, was the next possessor. He partially or totally rebuilt the manor house. From him this manor descended to Washington Shirley, 5th Earl Ferrers by whom it was sold in 1763 to Richard Grenville-Temple, 2nd Earl Temple. He was succeeded by his nephew, George Nugent-Temple-Grenville, 1st Marquess of Buckingham, and whose son, Richard Temple-Nugent-Brydges-Chandos-Grenville, 1st Duke of Buckingham and Chandos was also associated with the manor house. Biddlesden Abbey had possessions in the manor. By 1874, the manor house had been largely dismantled, some of the "inferior offices" becoming a farmhouse, and a "broad embattled tower" remaining.