By the middle of the 15th century the minor noble family of Horrenberg had died out. The prince-bishop of Speyer had to relinquish the authority of Rotenberg, to which Horrenberg belonged, to the Palatinate of the Rhine from 1462 to 1498. At this time the Horrenberg fell into ruin as the lords of the Horrenberg had no need of the castle and left the community to fend for itself. In the sources of the 15th and 16th century continuously refer to the "former castle".