There is an oral tradition that the whole area of the shrine was home of the Katsuragi clan and the Imperial Palace for the Emperor Suizei (Takaoka-no miya) was there.
神社一帯は葛城氏の本拠地で、綏靖天皇の皇居（高丘宮）があったという伝承が残る。