He came back to the post as Juichii Udaijin (Minister of the Right with Junior First Rank) after Nakamaro lost his position and was killed for his failure to drive Dokyo out of office (Rebellion of Fujiwara no Nakamaro) in 764.
天平宝字8年（764年）、仲麻呂が道鏡排斥に失敗して失脚し殺害された後（藤原仲麻呂の乱）、従一位右大臣として復帰した。