It is said that originally it was the residence of Imperial Prince Kaya of Emperor Kanmu, though the owner changed many times and finally in the early 11th century, Sessho (a regent) FUJIWARA no Yorimichi loved the place very much, owned, enlarged the premises to double in length to approx. 436m and constructed a gorgeous Shinden-zukuri (architecture representative of a nobleman's residence during the Heian period) buildings.
元は桓武天皇の皇子・賀陽親王の邸宅であったとされ、その後所有者は転々としたが、11世紀初頭に摂政藤原頼通は、この地を大いに気に入って敷地を倍の4町に広げて豪華な寝殿造の建物を造営した。