After that, Yoshitomo abandoned FUJIWARA no Nobuyori and went to Togoku to regain his power with his first son Yoshihira, his second son Asanaga, his third son Yoritomo, the families MINAMOTO no Yoshitaka (later Mutsu Rokuro Yoshitaka), Yoshinobu HIRAGA, and MINAMOTO no Shigenari (later Shigenari SADO), his followers Masakiyo KAMATA, Sanemori SAITO and Konnomaru SHIBUYA.
その後、藤原信頼を見捨て、長男義平・次男朝長・三男頼朝・一族の源義隆（陸奥六郎義隆）・平賀義信（平賀義信）・源重成（佐渡重成）、家臣の鎌田政清・斉藤実盛・渋谷金王丸らを伴い、東国で勢力挽回を図るべく東海道を下るが、