As is the case with Kasagi-dera Temple at the south of Kontai-ji Temple, there are many strange rocks and bizarre stones, and it is presumed that it developed as a place for mountain ascetic practices from ancient times; however, as is often the case with this sort of mountain temple, the circumstances of its development aren't clearly known.
南方に位置する笠置寺と同様、山内に奇岩怪石が多く、古くから山岳修行の地として開けていたと推定されるが、こうした山岳寺院の常として草創の経緯ははっきりわかっていない。