Yoshitoki, like his father and elder brother, became a Gokenin (immediate vassal of the shogun) by joining the army raised by MINAMOTO no Yoritomo in his early days, acquiring the position of Ie no ko (meaning a follower), a position that differed from that of an ordinary Gokenin in that someone holding this position would serve as a personal aide/bodyguard to the Yoritomo.
父や兄と同様に、早くから源頼朝の挙兵に参加して御家人となったが、一般の御家人とは区別される、頼朝の個人的な側近・親衛隊員である「家子」としての地位を獲得している。