His paternal grandmother was Ichijo, who was a nanny of Imperial Princess Muneko, an older sister of Emperor Goshirakawa who was born to the same mother; therefore, Yoshiyasu was close to Cloistered Emperor Goshirakawa and Josaimonin.
父方の祖母は後白河天皇の同母姉統子内親王乳母一条で能保は後白河法皇ー上西門院に近い立場にあった。