In "Genpei Seisui Ki" (Rise and Fall of the Minamoto and Taira clans), it is said that Yoshimori was from Ise Province and became the "closest retainer" of Yoshitsune following his encounter with Yoshitsune in Kozuke Province after being imprisoned and released from jail for killing his aunt's husband.
『源平盛衰記』では伊勢国出身で伯母婿を殺害して投獄され、赦免されて上野国で義経と出会い「一の郎党」となったという。