The three-storey castle itself has changed little since its completion in 1544, an incorporates elements from all of the three phases of building work in 1512–14, 1539–40 and 1543–44. It is now roofless but still standing up to tall, and covers , almost as big as the largest of the Device Forts at Deal in Kent. The first tower on the site was built from fine-grained yellow sandstone, with the later expansion making use of both yellow and grey sandstone, with imported Caen stone being used for the finer detailing. Ironstone, siltstone and brown sandstone rubble and boulders was used for the core of the castle walls, some of it taken from the local cliffs.