His father Yoshifumi was a son of TAIRA no Takamochi, a son of Imperial Prince Kazurawara (Takamochi renounced his membership in the Imperial Family and adopted the Taira clan), and had influence over Shimousa Province; Tadayori increased influence over Musashi Province and became opposed to his cousin TAIRA no Shigemori (son of TAIRA no Kunika).
父の良文は葛原親王の子平高望（臣籍降下し平姓を名乗る）の子で、下総国に勢力を持ち、忠頼は武蔵国に勢力を伸ばしたが、従兄弟の平繁盛（平国香の子）と対立した。