His father was assassinated in his childhood, following which incident his brother-in-law MINAMOTO no Tameyoshi took over as head of the family and he himself was placed under his uncle (his mother's brother) TAIRA no Tadamori's custody.
幼少期に父が暗殺され、義兄の源為義が家督を継ぎ、叔父(母の弟)の平忠盛の邸で育つ。