Yoshikata was staying at the residence of the Minamoto clan in Horikawa in Heian-kyo (the ancient capital of Japan in current Kyoto) but, in 1153, when MINAMOTO no Yoshitomo, who had gone to the Kanto region after becoming at odds with their father Tameyoshi, assumed a post in Shimotsuke Province and extended his power over the Southern Kanto region Yoshikata, at the father's command, went to the Northern Kanto region to retaliate against Yoshitomo.
平安京堀川の源氏館にいたが、仁平3年（1153年）父為義と不仲になり関東に下っていた兄の源義朝が、下野国に就任し南関東に勢力を伸ばすと、義賢は父の命により義朝に対抗すべく北関東へ下った。