Since he was a child of a concubine, his rank was put under his brothers Tokimune and Munemasa, who were the children of the lawful wife, as the third ranking by his father Tokiyori, and treated strictly and unkindly.
側室の子である為、父・時頼から序列三位として正室の子である弟の時宗や宗政よりも格下に置かれて厳しく冷遇された。