Probably in the first half of the 11th century, a new castle named "Kästenburg" (translated in the Palatinate dialect as the "chestnut castle") was built inside the former estate. It got its name because of the surrounding magnificent sweet chestnut forests. There is little known about its early history; there are speculations without any proof that it was founded as an Imperial Castle (Reichsburg) or that Emperor Henry IV had started his Walk to Canossa here in 1076. The only thing certain is that between 1090 and 1104 bishop Johann I of Speyer signed over the estate together with "Castle Meistersel" to the Bishopric of Speyer, which stayed the owner to the end of the 18th century.