Yang You was born in 605, as a son of Yang Zhao, the son and crown prince of Emperor Yang. His mother was Yang Zhao's wife, Crown Princess Wei. He was probably the youngest of Yang Zhao's three sons—he was clearly younger than Yang Tan (), born in 603, and probably younger than Yang Tong, whose birth year is unknown, based the order in which they were eventually created imperial princes. However, under Confucian principles of succession, he would have been considered Yang Zhao's proper heir, as his mother was Yang Zhao's wife, while Yang Tan's and Yang Tong's mothers were concubines.