KO no Moronao's family lineage can be traced back to the time when TAKASHINA no Koreaki, who is said to have been an illegitimate child (also said to have been their nanny's younger brother) of MINAMOTO no Yoshiie, the Minamoto clan's master builder, started to live with MINAMOTO no Yoshikuni, Yoshiie's third son, in Shimotsuke Province.
源氏の棟梁、源義家の庶子と云われる高階惟章（実際は乳母弟か）が、義家の三男源義国とともに下野国に住したことに始まる。