After the death of Yoritomo, Yoritaka returned to his territory, the estate of Wakatsuki, Shinano Province, and then received the rank of Jugoinoge (Junior Fifth Rank, Lower Grade) and was appointed as the Governor of Izu Province.
頼朝の死後、所領の信濃国若槻庄に下り、従五位下・伊豆守に叙せられている。