However, at that time, Yao's sons Yao Yí (姚彞) and Yao Yì (姚异) (note different characters and tones), who were both deputy ministers at this time, often received gifts from others, something that the popular opinion at the time criticized them and Yao Chong for. Meanwhile, Yao Chong's associate Zhao Hui (趙誨) was found to have accepted bribes from foreigners, a crime punishable by death but which Yao tried to save him from death for, displeasing Emperor Xuanzong. At that time, Emperor Xuanzong was set to declare a pardon for the criminals of Chang'an, but Emperor Xuanzong modified his pardon to specifically exclude Zhao, ordering that Zhao be caned 100 times and then exiled to the Lingnan region. Yao, in fear, offered to resign and recommended Song to succeed him. Emperor Xuanzong thus removed Yao and Yuan from their chancellor posts—with Yao receiving the honorific title "Kaifu Yitong Sansi" (開府儀同三司) but no other offices. Song and Su Ting became chancellors to succeed Yao and Yuan. In 717, after Yao made a comment that pleased Emperor Xuanzong, blaming the collapse of a hall of the imperial ancestral temple on the wood being too old rather than on divine displeasure, Yao became restored to Emperor Xuanzong's good graces and, while not made chancellor again, was allowed to meet Emperor Xuanzong once every five days.