From the mid 10th century until the mid 12th century, the counts of Bregenz ruled Maienfeld. Their castle at Maienfeld was destroyed in 1079 by Henry IV, HRE during the Investiture Controversy. The current Schloss Brandis was initially built in 1270–75 by the "von den Aspermont" family and later by Friedrich VII, count of Toggenburg. In 1465 it was rebuilt and expanded by the Baron von Brandis. The town was also fortified in the 13th century. A tall and thick section of wall is still standing today. The town walls enclosed an area of about . The walls were thickened and raised to in the 14th century and then, in a third phase, topped with thick crenallations and raised to .