When he was aged 3, his real father Sukeyasu KAWAZU was assassinated by a member of the family, Suketsune KUDO, and after that, together with his older brother Sukenari he was brought up by the Soga clan, the feudal load of Soga-so Estate in Sagami Province (present Odawara City, Kanagawa Prefecture) and the family into which his mother remarried, and he came to use the name of Soga clan.
3歳の時、実父河津祐泰が1176年（安元2年）に一族の工藤祐経に暗殺され、その後母の再嫁先である相模国曾我荘（現在の神奈川県小田原市内）の領主曾我氏において兄祐成とともに養育され、曾我氏を称した。