Later, Tomoyasu went away from Kyoto to Kamakura and worked under MIMAMOTO no Yoriie, who was the second Seii taishogun (literally, "great general who subdues the barbarians"), and became his close aide.
後に鎌倉へ下向して、2代征夷大将軍の源頼家に仕えて側近となる。