The image of Yoriie in "Azuma Kagami," edited by the Hojo clan who held the real power of Kamakura bakufu, describes him as a foolish ruler who pursued pleasures and engaged in adultery with his retainer's favorite concubine.
鎌倉幕府北条氏編纂である『吾妻鏡』における頼家像は、遊興にふけり家来の愛妾を寝取る暗君として描かれている。