However, amid the strong opposition to Tadakuni's reappointment from roju, Masahiro ABE and the former roju, Toshitsura DOI, Tadakuni was suspected of taking a bribe from Yozo TORII and Sanemon GOTO during the Tempo Reforms, which worked as a negative factor to Tadakuni and deprived him of 10,000 koku (2,780 cubic meters) from his additional territory and another 10,000 koku from his original territory as an punitive measure in September 1845, and his territory was reduced to 50,000 koku (13,900 cubic meters).
しかし老中・阿部正弘をはじめ、前老中・土井利位らは忠邦の再任に強硬に反対し、忠邦に対しても天保改革時代の鳥居耀蔵や後藤三右衛門らの疑獄の嫌疑が発覚し、弘化2年（1845年）9月に加増のうち1万石、本地のうち1万石、合計2万石を没収されて5万石となった。