He had a half sister, FUJIWARA no Shoshi who was the empress of Emperor Toba and biological mother of Emperor Sutoku and Emperor Goshirakawa, because of this relationship, he served closely to the retired Emperor Sutoku together with his younger brother, Saneyoshi TOKUDAIJI at first and countered Emperor Konoe's biological mother, Empress Dowager Bifuku and her party.
異母妹に鳥羽天皇中宮で崇徳天皇・後白河天皇生母の藤原璋子があり、その関係から当初弟の徳大寺実能とともに崇徳上皇に近侍し、近衛天皇生母の美福門院とその一派に対抗したとされる。