He also served Yoshiaki ASHIKAGA, a younger brother of Yoshiteru, and when Nobunaga ODA banished Yoshiaki from the capital, he and other vassals of the Muromachi bakufu (Japanese feudal government headed by a shogun) accompanied Yoshiaki all the way to Tomo, Bingo Province in the Chugoku region.
義輝の弟の足利義昭にも仕え、義昭が織田信長に追放された後も他の室町幕臣とともに義昭に従い、中国地方の備後鞆に下向した。