It is the site of the lost Sandown Castle. While undergoing construction in 1545, the fortification was attacked during the French invasion of the Isle of Wight when invaders fought their way over Culver Down from Whitecliff Bay before being repelled. The castle was built into the sea, prone to erosion and demolished fewer than a hundred years after it was built. In 1631, the castle was replaced by Sandham Fort built further inland. In 1781, the fort's complement consisted of a master gunner and over twenty soldiers. Sandham Fort was demolished in the mid-19th century and is now the site of Sandham Gardens.