After this incident, only his uncle Tadayoshi, the Dewa no kami was allowed to succeed to the family name; however, Tadayoshi's son, Tadatomo, succeeded as a close adviser to Ieharu, which allowed the family to regain the daimyo position in the Ohama Domain in 1768.
その後、叔父出羽守忠穀に家名存続のみが許されたが、その子出羽守忠友が家治の側近として活躍したため明和5年（1768年）大浜藩にて大名に復帰する。