However, when the Cloistered Emperor Goshirakawa was confined within his Toba-den (residence of Cloistered Emperor Goshirakawa), Narifusa, who was the close associate of Cloistered Emperor Goshirakawa was dismissed from his office and was exiled to Izu Province.
しかし治承3年（1179年）、平清盛のクーデタによって後白河法皇が鳥羽殿に幽閉されると、法皇の側近であった業房も解官の上、伊豆国に流罪に処された。