In 1548, however, Nagayoshi went over to Ujitsuna's side and Harumoto suddenly came to be in a disadvantageous position, and in 1549, he fled to the Omi Province with Yoshiharu ASHIKAGA and Yoshiteru ASHIKAGA after he was defeated by Nagayoshi at the Battle of Eguchi.
しかし天文17年（1548年）、長慶が氏綱側に寝返ったため、一転して晴元側が不利となり、天文18年（1549年）には江口の戦いで長慶に敗れて晴元は足利義晴・足利義輝と共に近江に逃亡した。