His father, Yoshitomo, was in the lineage of Emperor Seiwa, and was the head of the Kawachi-Genji, that established an influence in Togoku (the eastern part of Japan, particularly Kanto region) from time of MINAMOTO no Yorinobu, MINAMOTO no Yoriyoshi and MINAMOTO no Yoshiie, who headquartered in Kawachi Province.
父・義朝は清和天皇を祖とし、河内国を本拠地とした源頼信、源頼義、源義家らが東国に勢力を築いた河内源氏の棟梁である。