Meanwhile, he considered the new and rising power of the In-no-kinshin (vassal attending on a retired emperor) to be dangerous; there is a story told that he destroyed the palace of FUJIWARA no Akisue, saying that it was not up to his social position (according to "Kichibu Hikunsho").
一方で、新興の院近臣勢力に対しては警戒感を示し、藤原顕季の邸を身分不相応だとして破壊したという話が伝わっている（『吉部秘訓抄』）。