According to temple legend, the location of Saiho-ji Temple (西芳寺) was formerly the site of Prince Shotoku's villa, which Nara period monk Gyoki converted to a Hosso Sect temple known as 'Saiho-ji Temple (西方寺)' (a homophone of the current name) dedicated to the principal image Amida Nyorai.
寺伝では今、西芳寺のある場所は聖徳太子の別荘であったものを、奈良時代の僧・行基が寺にしたもので、当初は「西方寺」と称し、阿弥陀如来を本尊とする法相宗の寺であったという。