However, "Jingo-ji Temple ryakki" (a journal of Jingo-ji Temple), written around the period of the Northern and Southern Courts (in the middle of the 14th century) described that Jingo-ji Temple had the portraits including Goshirakawain, TAIRA no Shigemori, MINAMOTO no Yoritomo, FUJIWARA no Mitsuyoshi, and TAIRA no Narifusa and they were drawn by FUJIWARA no Takanobu; this was the clue to judge the figures as TAIRA no Shigemori, MINAMOTO no Yoritomo, and FUJIWARA no Mitsuyoshi.
しかし、南北朝時代(日本)初め（14世紀中葉）頃に成立したとされる『神護寺略記』に、「神護寺には後白河院、平重盛、源頼朝、藤原光能、平業房等の肖像があり、それらは藤原隆信の作品である」との内容の記述があり、これが、三幅の像主を平重盛、源頼朝、藤原光能に比定する根拠となった。