However, from the fact that Yorito bore no official title while his father, FUJIWARA no Masayori, was titled Jugoinoge (Junior Fifth Rank, Lower Grade), it is inferred that Yorito was demoted to Mutsu Province by the Imperial Court for his having taken Tadatsune's side during the revolt of TAIRA no Tadatsune and became an official of Taga no Kou (the provincial office in Mutsu Province).
しかし父親の藤原正頼が従五位下であったことと比較し頼遠が無官であることから、平忠常の乱において平忠常側についた頼遠が罪を得て陸奥国に左遷され、多賀国府の官人となったものと推測されている。