The manor was inherited in 1514 by Henry Daubeney, created Earl of Bridgewater for his services to Henry VIII. He spent much of his time in France and little time in Somerset. He may have begun the new house, but he went bankrupt and was involved in the disgrace of Catherine Howard, Henry VIII's fifth wife. It is unlikely that much, if any, of the current building was his work. The house was forfeit to the crown and then had various owners including Henry Grey, 1st Duke of Suffolk, who in 1552 sold it to William Clifton, a London merchant who had been assembling a Somerset estate. Clifton or his son John Clifton are thought to be responsible for most of the building at Barrington, possibly modeled on earlier work at Kirby Hall or East Anglian manors close to their roots at Kentwell Hall, Hengrave Hall or Channons Hall at Tibenham.