Candleston Castle, a fortified manor, was built in the 14th century. An enclosure, in the shape of a D, surrounded the structure. The late-medieval manor may have been constructed on the site of a previous building, based upon the presence of remnants of an earlier wall. The castle may have been built by Robert de Canteilupe III, who owned one ploughland at Candleston in 1320 according to the Despenser Survey of Glamorgan of that year. Robert III, who is not recorded after 1320, may have been succeeded by John de Cantilupe, who may have been the Abbot of Margam in 1315. Robert de Cantilupe III's daughter Joan, heiress of Candleston, married Sir William Horton of Tregwynt, Pembrokeshire.