When the Shinsengumi's promotion to Shogun's retainer was determined in July 1867, he decided to break away from the Shinsengumi with other members including Shimenosuke SANO, Tsukasa IBARAKI and Juro TOMIKAWA who had strongly believed the idea of Sonno Joi (reverence for the Emperor and the expulsion of foreigners), and asked them to join Goryo-eji (guards of Imperial mausoleums) formed by Ito who also separated from the Shinsengumi, however his request was turned down.
慶応3年6月に新選組の幕臣取り立てが決定すると、尊皇攘夷思想の強かった佐野七五三之助・茨木司・富川十郎らと共に脱退を決意して、伊東らが新選組と分離して結成していた御陵衛士への入隊を希望するが拒否されてしまう。