However, conflict arose over the agenda of Yokohama-ko Port against the background of different purposes between Yoshinobu HITOTSUBASHI who aimed to protect the Shogunate and the Satsuma clan who aimed to take advantage of the Council to gain leadership, therefore, in February 1864 (according to the old lunar calendar) Yamauchi went back to his domain, and in the following March other members also resigned, consequently the Sanyo-kaigi was broken up.
しかし、これを機に主導権を握ろうとする薩摩藩と幕府・一橋慶喜の思惑の違いが絡んで横浜鎖港をめぐって対立、1864年（元治元年）2月に山内が帰国、3月には残る全員が辞表を提出してあっけなく瓦解した。