When the Emperor Reigen began direct administration after the death of the Cloistered Emperor Gomizunoo, Sanetoyo fell into disfavor with him as "pro-bakufu clique" partly because the Emperor Reigen hated the bakufu and because he was in charge of bukedenso-yaku, and was ordered to be placed in confinement at home since 1683.
後水尾法皇崩御後、霊元天皇が親政を開始したが、霊元天皇は大の幕府嫌いであり、武家伝奏役にあったせいか、実豊は親幕派と睨まれ、天和(日本)3年（1683年）より蟄居を命じられた。