The house was first known to be owned in the 16th century by Edward Baynton (1517–1593) of Rowdon; he had previously been Battle Abbey Steward and had built Bromham Hall in 1538. "In 1538 Bromham was purchased from the Crown by the Battle Abbey steward, Sir Edward Baynton. He was one of the great Wiltshire landowners and built Bromham House. This was nearly as big as the royal palace of Whitehall and contained materials savaged from Devizes Castle and a royal manor house at Corsham." The diarist John Evelyn described the house as built in 1654; however, Sir Edward may have substantially built upon a previously existing structure, based on eyewitness evidence after the 1868 fire.