His father was originally from the Ota clan in Musashi Province, and moved to Oyama, Shimotsuke Province, taking a new surname, Koyama, and got married to SAMUKAWA no Ama, a second wife, who was a daughter of Munetsuna Hatta, and a wet nurse of MINAMOTO no Yoritomo.
父は武蔵国の太田氏の出であり、下野国小山市に移住して小山氏を名乗り、後妻に八田宗綱の息女で源頼朝の乳母である寒河尼を娶っていた。