Although he returned home he did not get along with his biological father, who was the new lord of the domain, and again he travelled, this time to Oki-no-erabujima island.
復帰するが、新藩主の実父で事実上の藩主の島津久光と折り合わず、再び、沖永良部島に流罪にあう。