However, it is said that when he inspected Nagasaki by the command of his domain in 1866, he met Shojiro GOTO and Ryoma SAKAMOTO there, and he realized that the expulsion of foreigners was impossible, so he gradually got into overthrowing the Shogunate.
しかし慶応2年（1866年）、藩命で長崎を視察したとき、ここで後藤象二郎や坂本龍馬と交わって、攘夷の不可なるを悟り、次第に倒幕へ傾いていったといわれている。