Although Shigemori was the first and real son of TAIRA no Kiyomori, he was not the son of the legal wife TAIRA no Tokiko who gave birth to TAIRA no Munemori and TAIRA no Tokuko, and therefore, he lacked any powerful support from his maternal relatives; since his wife was Keishi, a sister of FUJIWARA no Narichika, he became somehow isolated within the clan after the Narichika fell from power.
嫡男ではあったが正妻の平時子の子である平宗盛や平徳子とは母が異なり有力な外戚の庇護はなく、室が藤原成親の妹・経子であったため、成親失脚後は一門のなかでは孤立気味であった。