The details of his adolescence are unclear; according to common belief, he was from the Toki clan, a constable of Mino Province, and while serving Sengoku Daimyo Dosan SAITO, he took Dosan's side in the battle between Dosan and Yoshitatsu in 1556, in which AKECHI-jo Castle was attacked, resulting in the breakup of the clan.
青年期の履歴は不明な点が多いが、通説によれば、美濃国の守護土岐氏の一族で、戦国大名の斎藤道三に仕えるも、弘治(日本)2年（1556年）、道三と斎藤義龍の争いの際、道三方に味方し、義龍に明智城を攻められ一族が離散したとされる。