From narrative accounts of "Heike Monogatari" (Tale of the Heike) and the "Genpei Seisui ki" (Rise and Fall of the Minamoto and the Taira clans), the impression given of Yorimori is clearly that of a man who once joined the withdrawal of his clan from the capital, but then gave up on his clan, withdrawing from the strife of public life and relying on Yoritomo's mercy.
『平家物語』語り本系や『源平盛衰記』では、頼盛は一門の都落ちに同行しながら、頼朝の情けにすがり一門を見捨てた脱落者としての印象が強い。