Succeeding bishops held the manor until the reign of Henry VIII. Lords included William Giffard, Henry of Blois, Godfrey de Luci, John Gervais, Nicholas of Ely, John of Pontoise, John de Stratford, Cardinal Henry Beaufort, William Waynflete, and Richard Foxe. In 1531 William Cholmeley was appointed to be bailiff of Ivinghoe, which had come into the king's hands by the forfeiture of Cardinal Thomas Wolsey, who was Bishop of Winchester. It was, however, restored to the bishopric almost at once to Bishop Stephen Gardiner, and so remained until in 1551, when John Poynet, bishop, surrendered it to the King. In the following month Edward VI made a grant in fee of the manor to Sir John Mason (diplomat), kt., and Elizabeth his wife.