It was then bought by Sir William Fowle Middleton, 1st Baronet, whose son and heir, Sir William Fowle Middleton, 2nd Baronet, commissioned architect John Gandy-Deering to remodel it in the early 1830s. There was further remodelling of the building for Sir William between 1849 and 1855 by Sir Charles Barry, who also created the terraced gardens. Paine's central block was built in 3 storeys with a 5 bay frontage, to which Gandy-Deering added 3 further bays to either side.
It was subsequently acquired by Sir William Fowle Middleton, 1st Baronet, whose son and heir, Sir William Fowle Middleton, 2nd Baronet, commissioned architect John Gandy-Deering to redesign it in the early 1830s. There were additional renovations of the building for Sir William between 1849 and 1855 by Sir Charles Barry, who also created the terraced gardens. Paine's central block was constructed in 3 storeys with a 5 bay frontage, to which Gandy-Deering added 3 further bays on either side.