After it caught fire and was destroyed again in 1013, Michinaga transferred its ownership to FUJIWARA no Yorimichi, his heir, but Yorimichi was reluctant to reconstruct the building because he possessed a residence called Kaya-in, and therefore, the reconstruction work took thirty years and was completed at last in 1043, it was destroyed again in a fire in 1031 before its completion.
長和2年（1013年）に火災で再度焼失すると、道長は嫡男・藤原頼通に譲ったが、頼通は高陽院という居宅を有していたために再建には熱心ではなく、再建途中の長元4年（1031年）には再度焼失して完成したのは30年後の長久4年（1043年）であった。