Naosuke II who served as Tairo (chief minister) during the end of the Edo Period had spent his time at this castle town until he became the lord of the domain.
幕末における幕府の大老を務めた井伊直弼は、藩主となるまでをこの城下で過ごしている。