It is not known when Yu Di was born, but it is known that his family was from Henan Municipality (, i.e., the region of the Tang Dynasty eastern capital Luoyang). His family traced its ancestry to the prominent Xianbei clan Moniuyu () of Northern Wei, which changed its name to Yu when Emperor Xiaowen of Northern Wei had the Xianbei clans' names changed to Han names. Among Yu Di's ancestors were generals and officials of Northern Wei, Western Wei, Northern Zhou, Sui Dynasty, and Tang, including the prominent Northern Zhou general Yu Jin (). Yu Di's grandfather Yu Wang (于汪) served as the director of the archival bureau, while his father Yu Xiong () served as a military advisor to a prefectural prefect.