It is said that he could not succeed to the Mito Domain because he was treated as the eldest child out of wedlock by Yorifusa who had consideration about the feelings of his older brothers Yoshinao TOKUGAWA and Yorinobu TOKUGAWA who had no son.
頼房が兄である徳川義直・徳川頼宣に男子がいないのを憚って庶長子として扱ったため、水戸藩を継承することができなかったといわれている。