In 600, Yang Xiu's oldest brother Yang Yong the Crown Prince, over false accusations made by Yang Guang and Yang Guang's ally Yang Su, was deposed, and Yang Guang replaced Yang Yong as crown prince. Yang Xiu was displeased over the development, and Yang Guang was worried that Yang Xiu would make trouble for him. He instructed Yang Su to collect evidence of Yang Xiu's improper behavior and submit it to Emperor Wen. In fall 602, Emperor Wen summoned Yang Xiu back to the capital, sending the general Dugu Jie (獨孤楷) to replace him. Yang Xiu considered refusing to return to Daxing, but decided not to resist after being advised by Yuan Shimin (源師民) otherwise. However, even after Dugu Jie arrived at Yi Province, Yang Xiu had not gotten underway, and it took further urging by Dugu Jie for him to actually depart. Yang Xiu, after leaving Yi Province, considered returning and attacking Dugu Jie, but Dugu Jie, who had taken precautions, had his army in a state of high alert, and therefore Yang Xiu gave up on the idea. While Yang Xiu was still en route back to Chang'an, his mother Empress Dugu died.