Subsequently, the temple was used as a secluded residence of the retired like TAIRA no Yasuyori, a warrior who served Emperor Goshirakawa in the late Heian period, Saigyo, a poet in the late Heian period, and Tona, a poet in the Northern and Southern Courts period (Japan).
その後、この寺には平安時代後期の武士で後白河天皇に仕えた平康頼、平安時代後期の歌人西行、南北朝時代(日本)の歌人頓阿などが隠棲している。