The Imperial Prince's Palace belonged to Imperial Prince Sadazumi who was called Imperial Prince Momozono, later on the Palace was passed to MINAMOTO no Yasumitsu (Momozono Chunagon (vice-councilor of state)), FUJIWARA no Morouji (Momozono Dainagon (chief councilor of state)), FUJIWARA no Chikanobu, FUJIWARA no Koretada, after that it became FUJIWARA no Yukinari's (the Imperial Prince Yoshiakira's great grandchild) Palace and he built a temple within the Palace.
親王の邸宅は桃園親王と呼ばれた貞純親王の邸宅で、後に源保光(桃園中納言)、藤原師氏（桃園大納言）、藤原近信、藤原伊尹家へ移り、その後藤原行成(親王の曾孫にあたる)の邸宅となり、行成はその邸内に寺を建立した。