However, Emperor Konoe died childless in 1155 and Emperor Goshirakawa, who was supported by Tadamichi, was enthroned, leading to Yorinaga losing standing.
しかし、久寿2年（1155年）、近衛天皇が子なく崩御し、忠通の推す後白河天皇が即位すると頼長は失脚してしまう。