As the result, in a series of Ansei no Taigoku (suppression of extremists by the Shogunate) conducted by Naosuke II who was appointed Tairo (chief minister) in the same year, he was forced to resign in 1859 by an informal decision of the bakufu.
このため、同年大老に就任した井伊直弼の安政の大獄の一環で、幕府の内請により翌安政6年（1859年）辞官を余儀なくされた。