Since the Wakisaka family was fudai daimyo (a daimyo in hereditary vassal to the Tokugawa family), Yasuaya took a position responsible for the security of coastal defense of the Settsu Province as a member of the Sabaku-ha (supporters of the Shogun), and joined the First and Second conquest of Choshu in 1864 and 1866, respectively, as a member of the Edo bakufu army (the army of Japanese feudal government headed by a shogun).
脇坂氏は譜代大名であったため、安斐は佐幕派として摂津国の海防警備を務め、元治元年（1864年）の第1次長州征伐、慶応2年（1866年）の第2次長州征伐にも江戸幕府軍の一員として参加した。