The island seems to have been held, in association with the local manors of Uphill and Christon, by the Bek family, who granted it to Henry de Lacy, 3rd Earl of Lincoln. Although the mechanism is unclear, it next passed to the Berkeley family with Maurice de Berkeley, the second Baron Berkeley, holding it in 1315. The site was used again by warreners in the 14th and 15th centuries. They lived in one of the ruined priory buildings which was rebuilt.