This made the Cloistered Emperor Toba furious and in May 1155, Tameyoshi was dismissed from his posts of Saemon no daijo (Senior Lieutenant at the Left Division of Outer Palace Guards) and Kebiishi and handed over the family estate to his eldest son, MINAMOTO no Yoshitomo.
鳥羽法皇の怒りを買い、久寿2年（1155年）4月、為義は左衛門大尉と検非違使を解官され、家督を嫡子源義朝に譲った。