According to the legend written in "Tomomitsu Ko Ki" (biography of Tomomitsu Ko [Ko means His Serene Highness]), he was born, during Yoritomo's exile in Izu Province, to Yoritomo and the daughter of Samukawa no Ama who had been performing his mother's roll; after birth he was sent to the Hatta family, Samukawa no Ama's parents home, to be taken care of, then raised as the third son (some say the fourth son) of Masamitsu OYAMA and Samukawa no Ama; but there can be found utterly no mentioning to this story in the first rated history books of that time － such as the "Azuma Kagami" (The Mirror of the East) which was the official record of bakufu － therefore it is widely believed that this story is nothing but a mere supposition.
『朝光公記』によれば、伊豆配流中の頼朝の世話をしていた寒河尼の娘との間に生まれ、寒河尼の実家・八田氏へ預けられた後、小山政光と寒河尼の三男（四男説もある）として育てられたというのが、その伝説の筋であるが、幕府の公式記録『吾妻鏡』をはじめとする当時の一級資料には、一切、このことには触れられていないことから、推測の域を出ないというのが大方の見方である。