The shrine traces its origin to a Tendai temple built in the Asuka period (538 – 710) called Tōnomine-ji, built by the monk Jo'e (643 – 666). Jo'e was the oldest son of Fujiwara no Kamatari (614 – 669), founder of the Fujiwara clan. Jo'e located the temple on Tōnomine, a peak of on the southern side of Mount Goharetsu (). Jo'e moved the remains of Kamatari to a 13-story pagoda on the site. During the Heian period, the temple developed together with the prosperity of the Fujiwara clan. The emperors Daigo (884 – 930) and Go-Hanazono (1419 – 1471) attached special reverence to the temple, and bestowed it with various honorifics.