He was dismissed from all his posts when Yoshinaka staged a coup d'eta against the Cloistered Emperor, who had been on increasingly bad terms with Yoshinaka, but was reinstated to the post of Sessho when Yoshinaka was defeated in 1184.
義仲と法皇の仲が険悪になって、義仲によるクーデターが実行されたときには全ての任を解かれたが、1184年に義仲が討たれると、摂政に復した。