However after the destruction of the Hashiba clan in 1615, the Edo bakufu (Japanese feudal government headed by a shogun) revoked this name and confiscated the shrine's land, after which the shrine buildings fell into decay and the sacred object within the shrine was secretly relocated to Ima Hie-jinja Shrine.
しかし、1615年（元和(日本)元年）に羽柴氏が滅亡すると、江戸幕府により神号が廃され、社領は没収、社殿は朽ちるままにされ、神体は新日吉神宮にひそかに移し祀られた。