The construction of the original fortress was probably ordered by King Canute I, although this is not totally certain. He reigned 1167–1195 and had fortresses built on the Swedish east coast as defence against enemies from the other side of the Baltic Sea. During the 13th to 15th centuries, additions and changes were made. For example, new towers were built, and a new and thicker wall was constructed. The fortress was damaged on a number of occasions during these centuries, including in 1361, when King Valdemar IV (Atterdag) of Denmark attacked Borgholm.