However, the bakufu again accused him of insufficient removal of the repaired parts, and as a result, his territories in Aki and Bingo Provinces worth 500,000 koku were confiscated; instead he was given Takaino Domain of Takaino County, one of 4 counties in Kawanakajima of Shinano Province and Uonuma County, Echigo Province, worth 45,000 koku.
しかし破却が不十分であることを咎められ、安芸・備後50万石を没収、信濃国川中島四郡中の高井郡高井野藩、越後国魚沼郡、4万5000石に減封される。