The clan was incessantly beset by internal feuds like the one between his second son, MINAMOTO no Yoshikata, and his grandchild, MINAMOTO no Yoshihira, and once he was removed from office and forced to retire in 1154 (as stated earlier) and his first son, MINAMOTO no Yoshitomo, who had been appointed Governor of Shimotsuke Province, took over as the head of the family, other Minamoto clan leaders increasingly switched their support to the more capable and competent Yoshitomo.
一族についても、次男の源義賢と孫の源義平の対立等内輪揉めが絶えず、前述の久安6年（1154年）年の解官・隠居で既に下野守などに任官した長男、源義朝が家督を次ぐとますます多くの源氏与党の人心は彼から離れ、才幹豊かな義朝に靡いてしまう。