During the reign of Hidetada TOKUGAWA, he was chosen as fuyaku (a warrior who educates the Lord's son to become a great leader) of Tadanaga, but after Tadanaga's lands were forfeited and Tadanaga himself was ordered to commit suicide by disembowelment, he underwent hardship of being a ronin (a masterless samurai).
徳川秀忠の代に、忠長の傅役に選ばれるも、忠長が領地没収の上切腹させられた後は浪人の憂き目に遭う。