His father Shigeyori belonged to the Inui clan which was a branch of the Toki clan of Seiwa-Genji (the Minamoto clan), and he told his name as the Toki clan first and changed it to the Inui clan later.
父重頼は、清和源氏土岐氏の支流乾氏で、初め土岐氏を称しのち乾氏に改めた。