Therefore, he was once driven from a government post by MINAMOTO no Yoshinaka; however, he returned to his former post, Dainagon (chief councilor of state), in 1187 with confidence in him shown by Emperor Go-Shirakawa.
このため寿永2年（1183年）には源義仲によって一旦官職を追われるが、後白河天皇の信任を背景に文治3年（1187年）に元の地位である大納言に復帰。