The Duchess of Newcastle-under-Lyne died in 1822 while giving birth to twins, a stillborn daughter and a son, Lord George Pelham Clinton, who died 13 days after birth. In memory of her, the 4th Duke built a church and mausoleum at Milton, Nottinghamshire. In 1833, when the new church at Milton was finished, it became the parish church, replacing the old one at West Markham. This situation was reversed, however, in 1949 when All Saints, West Markham, was reinstated as the parish church, and the Duke's mausoleum was left to decay. Milton Church, now simply a mausoleum, was finally rescued in 1972 when the Churches Conservation Trust took it into guardianship.