Although Tadafusa's grandfather, Iwakuranomiya Prince Tadanari (the fifth prince of Emperor Juntoku), was regarded as the most likely candidate for succession to the Imperial Throne after the death of Emperor Shijo, he was defeated in the race to decide the next heir due to obstruction by the Kamakura bakufu (Japanese feudal government headed by a shogun) and was instead granted the reigning name of Iwakuranomiya.
祖父の岩倉宮忠成王（順徳天皇第五皇子）は、四条天皇崩御後に皇位継承の最有力者と目されながらも鎌倉幕府によって阻止されて、岩倉宮の宮号を賜る。