According to "Daihizan Bujo-ji Engi" (Legend of Bujo-ji Temple on Mt. Daihi) written by FUJIWARA no Michinori (also known as Shinzei), the temple was founded in 1154 at the end of the Heian period by Kanku Sainen (also known as Mitaki Shonin) under the order of Emperor Toba, with the Retired Emperor Toba's private Eleven-faced Thousand-armed Kannon statue installed as the principal image, and the Niomon gate and main hall constructed by FUJIWARA no Michinori and TAIRA no Kiyomori.
藤原通憲（信西）が記した『大悲山峰定寺縁起』によれば、この寺は平安時代末の久寿元年（1154年）、鳥羽天皇の勅願により観空西念（三滝上人）が創建したもので、鳥羽上皇の念持仏の十一面千手観音像を本尊として安置し、本堂や仁王門の造営には藤原通憲と平清盛が当たったという。