However, he was oppressed by Yoritomo who was afraid that Kai-Genji might increase their power, and in that situation, Ariyoshi's brother Tadayori was killed and Kanenobu was exiled to be overthrown.
しかし、甲斐源氏の勢威拡大を恐れる頼朝からの圧迫の中、兄弟の忠頼は誅殺され、兼信は配流され失脚する。