In 1862, he was accused of misgovernment during his years of service, and had to hand over the position of lord of the domain to his adopted son-in-law, Tadauji, and was driven to abdication of the headship of a family and suspension from the office.
文久2年（1862年）には在職中の失政を問われ、娘婿の忠氏に藩主の座を譲り隠居謹慎に追い込まれた。