Li Xiaogong was considered calm and knowledgeable in his youth. After Li Yuan rebelled against Emperor Wen's son Emperor Yang of Sui in 617 and captured the capital Chang'an, declaring Emperor Yang's grandson Yang You the Prince of Dai emperor (as Emperor Gong), Li Yuan made Li Xiaogong in charge of a campaign to persuade or capture the region south of the Qinling Mountains. Li Xiaogong was able to persuade some 30 commanderies to submit to Li Yuan, and he also defeated the rebel leader Zhu Can in battle, after which he, against the advice of his subordinates, refused to slaughter the captives, and it was said that because of this, more rebels were willing to surrender to him than before.