When Yoshitomo MINAMOTO, whom Hideyoshi followed, was defeated in the Heiji War in 1159, Tsunetaka, along with the Sasaki clan, escaped into the Kanto region and served Yoritomo MINAMOTO, who was the third son of Yoshitomo and had been banished to Izu.
平治元年（1159年）の平治の乱で父が従った源義朝の敗北により、一門と共に関東へと落ち延び、伊豆に流された義朝三男である源頼朝に仕える。