In 1824–25 George Child Villiers, 5th Earl of Jersey had the original village and manor house demolished to make way for him to expand Middleton Park eastwards. The castle mound and All Saints' church remain isolated within the extended park.
During 1824-25, George Child Villiers, 5th Earl of Jersey, razed the initial village and manor house to enlarge Middleton Park towards the east. The castle hilltop and All Saints' church were left as solitary remnants in the expanded park area.