Only thereafter did Yuchi began to fear Emperor Taizong and start to modify his behavior. In 637, as part of Emperor Taizong's scheme to bestow prefectures on his relatives and great generals and officials as their permanent domains, Yuchi's title was changed to Duke of E, and he was given the post of prefect of Xuan Prefecture (宣州, roughly modern Xuancheng, Anhui), to be inherited by his heirs. Soon, however, with many objections to the system, the strongest of which came from Zhangsun Wuji, Emperor Taizong cancelled the scheme, although Yuchi's title remained Duke of E. He subsequently served two different terms as commandant. In 643, he offered to retire, and the retirement was granted, but Emperor Taizong still requested that Yuchi attend imperial gatherings once every five days. Later that year, when Emperor Taizong commissioned the Portraits at Lingyan Pavilion to commemorate the 24 great contributors to Tang rule, Yuchi's was one of the portraits commissioned.