The last of the male line of Somery, John Somery, died in 1321. It is thought that the fortifications were complete by this date. The castle and estates passed to John Somery's sister Margaret and her husband John de Sutton. Subsequently, members of this family often used Dudley as a surname. John and Margaret were only in possession of the castle for a few years before the property was seized by the younger Hugh Despenser, a favourite of King Edward II. Despenser owned the castle from 1325-1326, being dispossessed when the king fell from power. The castle was returned to John and Margaret in 1327. It was probably during the time of John and Margaret's son and successor John Sutton II that a chapel and great chamber were added within the castle walls. Following the death of John Sutton II, the castle passed to his wife, Isabel, daughter of John de Cherleton who held it until her death in 1397.