Thereafter, he was left with SAMUKAWA no Ama, Yoritsuna's grandaunt and menoto (a woman who provides breast-feeding to a highborn baby) of MINAMOTO no Yoritomo (founder of the Kamakura bakufu), and became an adopted child of her husband, Masamitsu OYAMA.
その後、頼綱の大叔母で源頼朝の乳母あった寒河尼の元へ預けられ、その夫小山政光の猶子となった。