As Li Heng's ascension to be crown prince was against Li Linfu's wishes, and also because Wei Jian, who drew favor from Emperor Xuanzong due to his abilities to increase revenues for the imperial treasury, was becoming a rival to Li Linfu at court, Li Linfu looked for ways to incriminate both Wei and Li Heng. In 746, when the official Yang Shenjin (), at Li Linfu's instigation, reported to Emperor Xuanzong that, one night, Li Heng met Wei while both were sightseeing, and shortly after, Wei met a friend of his, the general Huangfu Weiming () at a Taoist temple, Li Linfu used these incidents to accuse Wei and Huangfu of secretly pledging allegiance to Li Heng and planning to remove Emperor Xuanzong to replace him with Li Heng. Wei and Huangfu were arrested and interrogated by Yang, Wang Hong (), and Ji Wen (), but Emperor Xuanzong, who did not want the case to further explode into a major incident although he believed Li Linfu's accusations, demoted Wei and Huangfu out of the capital and, for the time being, stopped the investigation. However, when Wei Jian's brothers Wei Lan () and Wei Zhi () submitted a defense of their brother Wei Jian later in 746—and the defense cited words from Li Heng—Emperor Xuanzong was incensed. Li Heng, in fear, divorced Crown Princess Wei and asked for the Wei brothers to be punished. The Wei brothers, along with a number of their associates—including the former chancellor Li Shizhi, Wei Bin (), Wei Jian's nephew Li Quan () the Prince of Xue, Pei Kuan (), and Li Qiwu (), were all demoted. (In 747, at Li Linfu's instigation, the Wei brothers, Huangfu, and Li Shizhi were all forced to commit suicide.)