The castle was built on a marsh by the village of Llanfaes. The French builders called the marsh "beaux marais" which means "beautiful marsh". This is how the town got its name. The people who lived village in Llanfaes were removed to a new village in the west of Anglesey to make way for the castle. This village was called Newborough and it still exists today.