However, in November 1196, he and his father were expelled from the Imperial Court and confined to their house due to a counter-move by TAKASHINA no Eishi and MINAMOTO no Michichika who were anti-Kanezane
しかし建久7年（1196年）11月、反兼実派の高階栄子と源通親らの反撃を受けて父とともに朝廷から追放され、蟄居することを余儀なくされた。