In 704, Yao, who was by that point "Fengge Shilang" (鳳閣侍郎), the deputy head of the legislative bureau of government (鳳閣, "Fengge"), citing the fact that his mother was old, requested to be relieved of his office so that he could attend to his mother. In summer 704, Wu Zetian made him Li Dan's secretary general, but allowed him to have the salary and prestige as officials of the third rank (i.e., chancellors—while the secretary general for an imperial prince was only ordinarily fourth rank, second division, first class). Less than a month later, she made him also the minister of defense (夏官尚書, "Xiaguan Shangshu") and chancellor "de facto" again, with the greater designation of "Tong Fengge Luantai Sanpin" (同鳳閣鸞臺三品). However, he declined the post of minister of defense, explaining that while he was serving Li Dan, for him to be in command of the army would hurt Li Dan (i.e., possibly bring suspicions on Li Dan). Therefore, Wu Zetian instead made him the minister of rites (春官尚書, "Chunguan Shangshu"). Around this time, Wu Zetian's lover Zhang Yizhi wanted to move 10 senior Buddhist monks from the capital Luoyang to a temple that he had built in Ding Prefecture (定州, roughly modern Baoding, Hebei). These monks did not wish to go and petitioned Yao, who ruled that the monks did not have to go, despite repeated requests by Zhang. Zhang therefore spoke to Wu Zetian against Yao, and Yao was demoted to be the minister of husbandry (司僕卿, "Sipu Qing"), although he remained chancellor. Soon, Wu Zetian, while letting Yao remain chancellor, made him the commanding general of the army at Lingwu, apparently again contemplating a campaign against Eastern Tujue. Before Yao departed for Lingwu, Wu Zetian met him personally and asked him to recommend someone who is suitable for being chancellor, and Yao recommended Zhang Jianzhi. She then made Zhang Jianzhi chancellor.