It is said that the residence was built by FUJIWARA no Yoshifusa who became the first sessho not belonging to an imperial family (sessho: a person who assumed the power of an emperor when the emperor could not do the necessary work because he or she was too young or had a serious illness).
人臣最初の摂政である藤原良房によって建てられたと伝えられている。