In 1624, when he did not follow Tadamasa's order to move Masanari's territory to Echizen, he became a masterless warrior again and was placed in confinement at home in the territory of his son Masakatsu.
寛永元年（1624年）、忠昌の越前転封に従わず再び浪人し、子の正勝の領内で蟄居を命ぜられる。