The castle was bought by Edward Seymour, the Duke of Somerset, in 1547; it passed shortly afterwards to John Dudley, the Duke of Northumberland, who sold it to the Crown in 1551. Shortly after this the prison in the castle diminished in size, vanishing entirely by the 1560s. The prison was recreated in the 1580s, however, for holding recusants, that is to say Roman Catholics who refused to attend Church of England services as was required by law. In 1595 Banbury Castle was leased to Richard Fiennes, the lord of Saye and Sele.