In 1661, as Banzan gained a reputation, the Tokugawa shogunate became wary of him once again, and in the end Chikasige MAKINO, the Kyoto Shoshidai (the governor of Kyoto appointed by the Tokugawa shogunate) at that time, purged him from Kyoto.
寛文元年（1661年）その名声が高まるにつれ再び幕府に監視されるところとなり、とうとう時の京都所司代牧野親成により京都から追放された。