In 1162, however he was dismissed by the will of the Cloistered Emperor Goshirakawa because he had been regarded as a close advisor of the Emperor Nijo.
しかし、二条天皇側近と見なされたことから、応保2年（1162年）に後白河法皇の意向で解官される。