With the fall of the Western Roman Empire and the subsequent Barbarian invasions, the flatlands were gradually abandoned as the inhabitants sought refuge in the hilly area which was more easily defended. By the middle of the 10th century AD the town was ruled by the prince-bishops of Belluno. In 1149, the Bishop of Belluno conferred the fief of Mussolente on Ezzelino I da Romano, and it remained in control of the Ezzelini family until their fall in 1260. It was then briefly under the control of Padua until 1262 when Treviso destroyed Mussolente's castle and took control of the town. In the early 14th century Mussolente's allegiance to the Scaligeri family made it a target of militias from other cities in the region who attacked and burnt down much of the town in 1320.