Yorimasa served the retired Emperor Toba and had connections with Bifukumon-in, wife of the retired Emperor, and FUJIWARA no Ienari, a close aides to the retired Emperor.
頼政は鳥羽院に仕え、寵妃の美福門院や院近臣の藤原家成と交流を持っている。