After that, he successively served as Bizen no Gon no kami (Provisional Governor of Bizen Province) and Sakone no chujo (Middle Captain of the Left Division of Inner Palace Guards), but suffered economic hardships and went to Suruga Province in 1540.
その後、備前権守・左近衛中将を歴任したが、経済的に困窮し、天文9年（1540年）より駿河国に下った。