Goryo-eji broke ranks with Shinsengumi because they were Kinno-Tobaku, people loyal to the emperor with an aim to overthrow the shogunate government, and Sabaku, the Bakufu supporters; which allegedly caused Goryo-eji to always carry swords with him while he slept in case he was ever raided by the Shinsengumi.
新選組とは佐幕と勤王倒幕で袂をわかっただけに、新選組の襲来を恐れていつも刀を抱いて寝たという。