It is said that FUJIWARA no Morosuke (a nephew of FUJIWARA no Tokihira and the son of FUJIWARA no Tadahira who is said to be a relative of the Sugawara clan) later donated the buildings of his residence to rebuild the magnificent shrine pavilions.
後に藤原師輔（藤原時平の甥であるが、父の藤原忠平が菅原氏と縁戚であったと言われる）が自分の屋敷の建物を寄贈して、壮大な社殿に作り直されたと言う。