After his father was defeated and killed in the Heiji Disturbance, he was confined in Kurama-dera Temple, but later went down to the town of Hiraizumi, Mutsu Province to seek refuge under the protection of FUJIWARA no Hidehira, head of the Oshu-Fujiwara clan.
平治の乱で父が敗死したことにより鞍馬寺に預けられるが、後に陸奥国平泉町へ下り、奥州藤原氏の当主藤原秀衡の庇護を受ける。