In 1350, Otto von Pienzenau became by marriage Lord of Zinneberg. The castle's ownership then passed in 1596, once again by marriage, to the Fugger family, in whose hands it remained for the next 230 years. In 1632, during the Thirty Years' War, Schloss Zinneberg was burnt down. At this time arose the legend that a secret passage, allegedly still existing today, between the castle and the town afforded some of the castle's inhabitants an escape to safety. Already by 1640, Count Johannes Friedrich, himself a Fugger, had the castle rebuilt. In 1825, the castle was bought by Bavarian Electress Maria Leopoldine, who was married to Count Ludwig of Arco. She had it remodelled to her own tastes, and the castle is still mostly in this form today. From 1850 to 1868, the castle was owned by the Marquis Fabio von Pallavicini, who in turn sold it to Friedrich Wilhelm Scanzoni von Lichtenfels. In 1898, became the new lord of the castle. He was responsible for further expansions to the building, which still lend the castle a feudal character today.