When the confrontation between MINAMOTO no Yoritomo and MINAMOTO no Yoshitsune turned serious after the collapse of TAIRA clan, he was censured by Yoritomo, along with TAKASHINA no Yasutsune and TAIRA no Tomoyasu, for supporting Yoshitsune and was removed from office once again.
平氏滅亡後、源頼朝と源義経が対立を深めると、高階泰経、平知康らとともに義経支持派として頼朝から弾劾され、再び解官の憂き目に合っている。