However, Yoshiharu's relationship with Harumoto never stayed calm; on the contrary, it was always unstable as the shogun confronted his greatest supporter, lost in a battle against him, escaped to Omi Sakamoto in 1541, made peace with him and returned to Kyoto in 1541.
しかし、その後も晴元と対立して敗れたのち、和解して帰京するといった行動を繰り返しており、天文10年（1541年）には近江坂本に逃れ、天文11年（1542年）には京都へ帰還。