During the German Peasants' War in 1525 the Kästenburg was occupied and looted, but not destroyed, by the "Nußdorfer Bauernhaufen" (literally translated: Nußdorf peasant bunch). In 1552 it was conquered and burned down by troops of Margrave and mercenary-leader Albrecht Alkibiades of Brandenburg to whom a tribute of 150,000 gulden was denied. Bishop Marquard of Speyer, who was in office from 1560–81, only arranged a very provisional rebuilding of the residential buildings and made the ruin the seat of a forester.