Found guilty of adultery with an Imperial court lady around February or March 1607, he was punished and temporarily expelled from Kyoto by the Imperial order of Emperor Goyozei, but after returning to Kyoto, his loose moral was not rectified and he continued having illicit liaisons with court ladies, taking many high court nobles out with him.
慶長12年（1607年）2月、宮廷女官との密通が明らかとなった結果、後陽成天皇から勅勘を蒙って追放されるが、のち京へ戻った後も素行は収まらず、多数の公卿を誘っては女官と不義密通を重ねた。