Regarding his relationship with Shimokobe clan, which is defined as a group of samurai in the eastern part of Japan, it is believed that the master-retainer relationship was established in the background of the establishment of Shimokobe-no-sho after Yorimasa went to the Shimosa Province together with his father Nakamasa who was appointed as Shimosa no kami (Governor of Shimosa Province).
東国武士である下河辺氏との関係については、下総守に任じられた父仲政が頼政を伴い任国に下った際、下河辺荘の成立を背景として主従関係を結んだと考えられている。