However, since he supported Kobu Gattai (Union of Court and Camp) movement led by the Satsuma domain and approved 'Kokai enryaku saku' (a strategy to trade foreign countries in order to strengthen the power of state), he was regarded as an enemy by supporters of the doctrine of restoring the Emperor and expelling the barbarians.
しかし、薩摩藩の主導する公武合体運動を支持して「航海遠略策」に賛同したため、尊皇攘夷派の志士から敵視された。