He was brought back to the Tokugawa family by Tsunashige, who had no male children as heir, when he was nine years old, and when he reached adulthood he went by the name of Tsunatoyo after being granted use of a portion of the real name of uncle Ietsuna TOKUGAWA, the fourth shogun, and after his father died in 1678, he took over as head of the family at age 17 and was raised by his grandmother JUNSEI-IN.
9歳のとき、他の男子に恵まれなかった綱重の世嗣として呼び戻され、元服して伯父である4代将軍・徳川家綱の偏諱を受けて「綱豊」と名乗り、延宝6年（1678年）の父の死後、17歳で家督を継承し祖母・順性院に育てられた。