Although he was trusted by Cloistered Emperor Goshirakawa as his closest associate, he came under constant criticism from nobles including Kanezane KUJO, his uncle, due to the close relationship he and his family had maintained with the Taira clan, and in March 1186, he was eventually dismissed from all his posts and forced into retirement as he was regarded as the person who had advised the Cloister Emperor to issue a decree that ordered MINAMOTO no Yoshitsune to defeat his elder brother, MINAMOTO no Yoritomo.
その後は後白河法皇随一の側近として信任を受けたが、かつて平家と親しかった経緯から叔父である九条兼実をはじめとする公卿から基通に対する非難が相次ぎ、1186年3月には源義経が兄・源頼朝追討の院宣を法皇に出させることを仲介した張本人と見なされて全ての任を解かれ、篭居されるに至った。