TAIRA no Masamori and TAIRA no Tadamori of the Ise-Heishi (Taira clan), who had a connection with Insei, or the administration of the Retired Emperor, in the days of Yorimasa's father Nakamasa, gradually increased their power; the Kawachi-Genji (Minamoto clan) (the family line of MINAMOTO no Yorinobu, brother of Yorimitsu), who had begun to increase power through their connection with the Sekkan regent families, repeated internal strife among themselves and eventually decreased their power.
父仲政の頃から院政と結びついた伊勢平氏の平正盛、平忠盛が次第に台頭し、一方、摂関家と結びつき台頭していた河内源氏（頼光の弟源頼信の系統）は同族の内紛を繰り返して勢力を後退させている。