When his grandfather MINAMOTO no Tameyoshi and his uncle MINAMOTO no Yoshitomo were each other's enemies, MINAMOTO no Yoshikata received an order from his father, Tameyoshi, to go to Kanto from his family's base in Kyoto; Yoshinaka was born in Okura Yakata mansion in Musashi Province (present-day Ranzan-machi, Hiki-gun, Saitama Prefecture) as the second son of Yoshitaka.
祖父の源為義と伯父の源義朝が対立的関係になった時、父為義の命を受けて本拠地の京都から関東に下った源義賢の次男として武蔵国の大蔵館（現在の埼玉県比企郡嵐山町）に生まれる。