According to the "Heike monogatari," it was because he recited a waka (a 31-syllable classical Japanese poem) lamenting his obscure status that, when Kiyomori learned of it, made him realize 'I forgot about Yorimas' and therefore, he recommended him for promotion.
『平家物語』では、不遇の身を嘆く和歌を詠み、それを知った清盛が、「頼政を忘れていた」と推挙したことになっている。