The reason why nyoingo (a title of respect given to close female relatives of the Emperor or a woman of comparable standing) Kayanoin was bestowed on FUJIWARA no Taishi (Yasuko) (a great-great-granddaughter of Yorimichi and a daughter of Kanpaku (chief adviser to the Emperor), FUJIWARA no Tadazane) is derived from the fact that she lived in this palace.
鳥羽天皇の妃であった藤原泰子（頼通の玄孫、父は関白藤原忠実）に高陽院の女院号が与えられたのも、ここに居住していた事に由来する。