When Yoshitaka was killed by his nephew, Yoshihira in 1155, he went down to Shinano Province to take revenge, and invaded Emperor Toba's territory.
久寿2年（1155年）、義賢が甥源義平に討ち取られると、その復仇を果たすべく信濃国に下向、鳥羽天皇領を侵犯する。