In his later years, he lived with his lover Abutsuni, who wrote "Izayoi Nikki" (The Diary of the Wanning Moon) and doted upon her son Tamesuke REIZEI, leaving issues upon property inheritance.
晩年は「十六夜日記」を記した阿仏尼と同棲してその子冷泉為相を溺愛し、遺領相続に関して問題を残した。