However, with his relationships with Shogun Yoshizumi and Takakuni HOSOKAWA gradually worsening, and Tsunehisa AMAGO of Izumo province invading his estates of Iwami province and Aki province, he retired from the post of Kanrei-dai and returned to Yamaguchi.
しかし次第に将軍・義稙や細川高国と不仲になり、さらに領国の石見国や安芸国に対して出雲国の尼子経久が侵攻を開始してきたため、永正15年（1518年）8月2日に管領代を辞して山口に帰国した。