Choshu clan, however, raised the Kinmon Incident in 1864 and became Emperor's enemy so that he was punished for being in touch with Choshu clan and sent to domain's territory to be confined.
しかし、翌元治元年（1864年）長州藩が禁門の変を起こし朝敵となると、長州藩に通じたとして処罰され、藩地へ送られて幽閉された。