The legend of the temple says that this was originally a hermitage of the Buddhist priest Genpin, who did not like earthly affairs and moved to the foot of the mountain called Miwa, despite the fact that he was trusted greatly by the Emperor Kanmu and the Emperor Saga.
桓武天皇・嵯峨天皇に厚い信任を得ながら、俗事を嫌い三輪山の麓に隠棲したという玄賓（げんぴん）僧都の庵と伝えられる。