It was then bought by Sir William Fowle Middleton, 1st Baronet, whose son and heir, Sir William Fowle Middleton, 2nd Baronet, commissioned architect John Gandy-Deering to remodel it in the early 1830s. There was further remodelling of the building for Sir William between 1849 and 1855 by Sir Charles Barry, who also created the terraced gardens. Paine's central block was built in 3 storeys with a 5 bay frontage, to which Gandy-Deering added 3 further bays to either side.
Later, it was purchased by Sir William Fowle Middleton, 1st Baronet, whose son and successor, Sir William Fowle Middleton, 2nd Baronet, commissioned architect John Gandy-Deering to renovate the building in the early 1830s. Subsequent renovations of the building for Sir William took place between 1849 and 1855 by Sir Charles Barry, who also designed the terraced gardens. Paine's central section was built in 3 storeys with a 5 bay facade, to which Gandy-Deering added 3 additional bays on either side.