He did not have a close relationship with TAIRA no Shigeko, who was Emperor Takakura's birth mother as well as TAIRA no Kiyomori's wife, TAIRA no Tokiko's younger sister, also, he was not close to his father, Emperor Goshirakawa.
また、高倉天皇生母で平清盛の妻平時子の妹平滋子と不仲で、父後白河とも疎遠であったといわれている。