After the sentence of the sanction, Toyonao went under the protection of Korenori KAMEI who was the Lord of Shikano-jo Castle and hired Toyonao by paying him a stipend of 500 koku, thereby Toyonao moved to Shikano, finally moving again to Kyoto to follow the Kamei clan who was transferred by the shogunate (which was called 'tempo' in Japanese.)
その後は鹿野城の亀井茲矩に保護を受け、500石で召抱えられ鹿野へ移住したが、亀井氏の転封で京都に移住した。