Although there was no sufficient historical evidence about Yorimasa in his youth, it was recorded that he went to Shimosa Province together with his father Nakamasa when Nakamasa was assigned to govern that province.
青年期の頼政について史料は乏しいが、父の仲政が下総国受領に赴任した時に、これに同行している。