According to a heresy, he was a son of MINAMOTO no Yorizane (Settsu-Genji [Minamoto clan]), but there is a slight problem in their ages when considering the period in which Yorizane was active and the birth year of his elder brother Yoriyoshi.
異説では源頼実（摂津源氏）の子であるともされるが、頼実の活動時期に兄頼義の生年なども考慮に加えると世代的に幾分無理が生じる。