In 1180, after his brother, Yoritomo, raised an army, there is no historical material showing when Noriyori joined his brother; however, it would appear that he acted in cooperation with Kai-Genji (Minamoto clan) in his birth place, Totomi Province.
治承4年（1180年）に挙兵した兄・頼朝の元にいつ参戦したかは明示した史料はないが、最初は頼朝ではなく、出身の遠江国を中心に甲斐源氏などと協力して活動していたと考えられる。