His father Kagetoki, who fought for the Taira clan, saved Yoritomo's life, and later when Yoritomo took up arms for the second time and entered Kamakura to take control of the Kanto region, Kagetoki served to Yoritomo and became a gokenin (an immediate vassal of the shogunate in the Kamakura and Muromachi through Edo periods) and was given important posts.
平家方で参戦していた父の景時は頼朝の命を救い、後に頼朝が再挙して鎌倉に入り、関東を制圧すると景時は頼朝に臣従して御家人に列し、重用された。