Nariyori's mother was the daughter of FUJIWARA no Toshitada of Gon Chunagon, and Nariyori was the younger half brother of FUJIWARA no Mitsuyori and of FUJIWARA no Korekata.
母は権中納言藤原俊忠の娘で、藤原光頼、藤原惟方の同母弟に当たる。