After Lord Charles Spencer completed the purchase of the manor in 1770, he seems to have had Wheatfield Park enlarged with the addition of a bow-fronted north wing. By 1787 there was an ornamental lake in front of the house. The house caught fire on New Year's Day 1814. Fire engines from Shirburn and Watlington fought the blaze, but were hampered by a severe frost that had frozen much of the available water. This was during the Napoleonic Wars, and French prisoners of war on parole from Thame came to help. Their French officer helped to organise the villagers to fight the fire, but when he proposed that half of the house could be saved by blowing up the other half, Lord Charles Spencer refused stating that he feared for the safety of the crowds of people and of St. Andrew's church.