When Naosuke II, who had been having a relationship with him and took over the lord of domain, assumed the post of Tairo (chief minister) of bakufu (Japanese feudal government headed by a shogun) in 1858, he became koyonin of the domain.
かねてより親交のあった井伊直弼が藩主を継ぎ、安政5年（1858年）直弼が幕府大老に就任すると、自身は藩の公用人となる。