John Seymour, 4th Duke of Somerset (1629–1675), uncle, inherited the estate in 1671 on the death of the 3rd Duke, and in 1672 he rebuilt Totnam Lodge and redesigned the deer park, which at that date included long tree-lined walks and a deer "chase". He died in 1675, aged 46, only three years after having started the rebuilding. Being childless and faced with the Dukedom passing by law to his first cousin once removed and heir male the 5th Duke, who was seated at Marlborough Castle in Wiltshire, he bequeathed the unentailed Seymour estates to his niece Elizabeth Seymour, the wife of Thomas Bruce, 2nd Earl of Ailesbury (1656–1741), and thus the Seymour estates passed to the Bruce family.