There was an agreement between Goryo-eji and Shinsengumi that they would prohibit the exchange of members in the first place, but the previously mentioned ten were ignorant of this fact and attempted to desert Shinsengumi to join Goryo-eji, thus they have no place to go at all.
そもそも御陵衛士と新撰組との間には隊士の行き来を禁止する約束があり、そのことを知らずに新撰組を脱走して御陵衛士に加わろうとした彼らは行き場所を失った形となったのだ。