In 1554, Toshimasa (Dosan) passed on his family estate to his son, Yoshitatsu SAITO, shaved his head at Jozai-ji Temple to enter Buddhist priesthood, named himself Dosan and went into retirement at Sagiyama-jo Castle.
天文23年（1554年）、利政は家督を子の斎藤義龍へ譲り、自らは常在寺で剃髪入道を遂げて道三と号し、鷺山城に隠居した。