The castle is often featured as the stage for mysterious story because the previous lord of the domain and potential lords had died prematurely before Naosuke II took the position.
井伊直弼が藩主の座に就くまでに先の藩主やその候補者の多くが夭折（ようせつ）していることから、神秘的な物語の舞台に採り上げられることが多い。