When Toba villa (Jonan villa) was built by the Emperor Shirakawa, the shrine became part of the estate and was frequently visited by successive emperors and retired emperors.
白河天皇が鳥羽離宮（城南離宮）を造営してからはその一部とされ、代々の天皇、上皇がしばしば行幸した。