From this time Harusue began attaining power within the Imperial Court yet, as he had his daughter to merry kanpaku Hidetsugu TOYOTOMI, who was Hideyoshi's nephhew and his adopted son of Hideyoshi, in September 4, 1595, Hidetsugu was forced to commit suicide on Mt. Koya over suspicion of his fomenting rebellion, with his relatives, wife, and mistress being killed, and through this connection Harusue was forced to step down from his appointed office and was banished to Echigo Province.
これにより晴季は朝廷内で重きをなしたが、秀吉の甥で養子の関白豊臣秀次に娘を嫁がせていたため、文禄4年（1595年）8月に秀次が謀反の疑いを懸けられ高野山で自害を強いられ、秀次の一族妻妾が殺されると、これに連座し、越後国に流罪となり失脚した。