This might be because he was not a person of character and then, treated coldly by people around him, or he was affected by the fall of his father's second cousin and his boss, FUJIWARA no Nakamaro.
これは、名足が人格者ではなく周囲から疎んぜられていたためか、または父の又従兄弟で上司でもあった藤原仲麻呂の失脚の影響したのではないかと見られている。