Meanwhile, Zhang Shuo and Zhang Jiazhen had not been on good terms, because Zhang Shuo, who ranked below Zhang Jiazhen in the precedence of chancellors, had previously served as Zhang Jiazhen's superior while serving as deputy minister of defense, and therefore resented Zhang Jiazhen for not showing him more respect. Meanwhile, Zhang Jiazhen's brother Zhang Jiayou had been made a high-ranked general, but in 723 was accused of corruption. At Zhang Shuo's suggestion, Zhang Jiazhen wore plain-colored clothes and waited outside the palace for punishment, to show remorse. Instead of escaping punishment as Zhang Jiazhen hoped, however, Zhang Jiazhen, without getting another audience with Emperor Xuanzong, was demoted to be the prefect of You Prefecture (幽州, roughly modern Beijing). Zhang Shuo then replaced him as "Zhongshu Ling", drawing hatred from Zhang Jiazhen, who commented, "There can be two people serving as "Zhongshu Ling". Why did he have to squeeze me out?" In 724, Zhang Jiazhen was recalled to serve as minister of census (戶部尚書, "Hubu Shangshu"), as well as secretary general at Yi Prefecture (益州, roughly modern Chengdu, Sichuan). Emperor Xuanzong had Zhang Jiazhen join the chancellors for a banquet. Zhang Jiazhen, still resenting Zhang Shuo, cursed Zhang Shuo when they were both at the banquet, requiring Yuan Qianyao and Wang Jùn to step in to moderate between the two.