However, with Fujiwara no Nakamaro's increased influence, he was demoted, in 745, to become betto (head) of Chikushi Kanzeon-ji Temple, his fubutsu (products of the fuko) were confiscated, and in the following year of 746, he passed away at his post.
しかし、藤原仲麻呂が勢力を持つようになると、745年（天平17年）筑紫観世音寺別当に左遷、封物も没収され、翌746年（天平18年）任地で没した。