Later, he also accompanied Yoritomo who was going up to Kyoto in December 1190, however, in January 1191 (the next month) when Yoritomo was to return to Kanto District, he suddenly ran away on the very day of his departure, without telling his whereabouts even to his Kenin (retainers).
さらに建久元年（1190年）11月に頼朝の上洛に随行するが、翌月頼朝が関東に下向するその日の朝、家人にも行方を知らせずに突然逐電した。