Yoritomo was on close terms with Jien, a monk of the Tendai sect in the early Kamakura period, and composed waka (a traditional Japanese poem of thirty-one syllables), which his zotoka (exchange of poems), 'I can ill afford to understand to hold back without a word, please do exhaust your whole heart in the letter as if you traverse to Tsubo no ishibumi (stone monument) at the end of Mutsu,' was selected to Shinkokin Wakashu (New Collection of Ancient and Modern Japanese Poetry).
慈円と親交があって和歌を詠み、贈答歌の「陸奥のいはでしのぶはえぞしらぬふみつくしてよ壺の石ぶみ」は新古今和歌集に入撰している。