Although he was originally Kenin (a retainer) of TAIRA no Tomomori, he gave up on the Taira family in December 1180, and came to serve MINAMOTO no Yoritomo by an intermediation of Nagakiyo KAGAMI (加々美長清) who was his peer.
元々平知盛の家人であったが、治承4年（1180年）12月に平家を見限り、同僚であった加々美長清の仲介で源頼朝の麾下に入る。