He was the son of MINAMOTO no Yoriyuki, who was the second son of MINAMOTO no Nakamasa, but, as his father committed suicide at a young age, he was adopted by his uncle, MINAMOTO no Yorimasa.
源仲政の次男源頼行の子として生まれるが、父が若くして自害したため、伯父源頼政の養子となる。