In 1173, a year before his death, he was dismissed because he took the blame for a fire that occurred in Tonomine, and reportedly he went into retirement in Eikyu-ji Temple.
死の前年に当たる承安3年(1173年)には多武峰の火災の責任を問われて解官され、その永久寺に退隠したと伝えられる。