After his father and his oldest brother died during the Heiji War in 1159, he was taken to the Imperial Court by his maternal uncle, FUJIWARA no Noritada, in Kanuki in Suruga Province (present day Kanuki-cho, Numazu City, Shizuoka Prefecture).
平治元年（1159年）の平治の乱で父兄が死亡した後、駿河国香貫（現静岡県沼津市香貫町）にて母方の伯父の藤原範忠によって朝廷に差し出された。