His father, Yorimasa, supported TAIRA no Kiyomori in the Heiji Rebellion, and as a result, the Kawachi-Genji (Minamoto clan) led by MINAMOTO no Yoshitomo (in the family line of MINAMOTO no Yorinobu) was destroyed while the Settsu Genji remained an important family under the Taira administration.
父の頼政は平治の乱では平清盛に味方し、その結果、源義朝の河内源氏（源頼信の系統）は没落するが、摂津源氏は平氏政権下の中央に留まることになった。