The Toganoo area in which Kozan-ji Temple is located is even deeper in the mountains than Takaosan Jingo-ji Temple - known for its red autumn leaves, has long been a place of mountain asceticism and is thought to have once been the site of many small temples.
高山寺のある栂尾は、紅葉の名所として知られる高雄山神護寺からさらに奥に入った山中に位置し、古代より山岳修行の適地として、小寺院が営まれていたようである。