Out of consideration for Yorimichi's lawful wife and the importance of succeeding the Sekkan-ke, he was nurtured at the Tsuchimikado-dono Palace of FUJIWARA no Michinaga (Father of Yorimichi and grandfather of Michifusa) who was the virtual head of the Sekkan-ke.
また、頼通の正室への配慮と摂関家を継ぐという重要性から、摂関家の事実上の長である藤原道長（頼通の父・通房の祖父）の土御門殿で養育される。