He was punished by being deprived of his fief due to the false charge by Hachizaemon BABA who became a ronin (masterless samurai) in the Incident of Daihachi OKAMOTO (1612) and the Incident of Nagayasu OKUBO (1614) (there are various theories on this and described later) and he was exiled to the Omi Province and put under the control of Naotaka II.
慶長17年（1612年）の岡本大八事件、慶長19年（1614年）の大久保長安事件による浪人の馬場八左衛門の讒言により（諸説あり、後述）改易され、近江国に配流され、井伊直孝に預けられる。