Then Nobunari NAITO followed by Nobumasa NAITO became the lords of the castle, but it was torn down in 1615 and most of the material was used for building the Hikone-jo Castle.
内藤信成・信正が城主になるが1615年（元和(日本)元年）に廃城になり資材の大半は彦根城の築城に流用された。