The house was owned by Bayntuns since the 15th century. Sir Edward Bayntun (1517–1597) built the new house circa 1554, and married a Howard relative Agnes Ryce (d. 1574), formerly the mistress of a nobleman Lord Stourton (d. 1548). Their descendants held the house and estate until 1864 when both passed out of the family. Sir Edward was a Member of Parliament for Wiltshire County in 1562-1563 and again in 1570; M.P. for Devizes in 1571; M.P. for Calne in 1572 and M.P. for Chippenham Borough in 1588-1589 and High Sheriff of Wiltshire in 1570. His first wife's mother Lady Katherine Howard (d. 1554), was a half-aunt of one of Henry VIII's wives – Anne Boleyn and stepmother of another, Katherine Howard; she was daughter of the 2nd Duke by his second wife Agnes Tilney; Anne Boleyn and Katherine Howard were his granddaughters by his first wife Elizabeth Tilney (cousin of Agnes). However, her husband was Rhys FitzUryan.