Probably because he was on bad terms with his older brother, he left the Hoshina clan in 1622 and, after wandering provinces, stayed with his maternal uncle Sadakatsu MATSUDAIRA, the third son of Toshikatsu HISAMATSU and the lord of the Kuwana Domain in Ise Province.
また、兄とは仲が悪かったらしく、元和8年（1622年）には保科氏を去って、諸国を放浪した後に母方の叔父である伊勢国桑名藩主・松平定勝（久松俊勝の3男）のところに身を寄せた。