In 1687, the Tokugawa shogunate directed Tadayuki MATSUDAIRA (the feudal lord of the Koga Domain of Simousa Province and the heir of Nobuyuki MATSUDAIRA) to take Banzan (then aged 69) into custody and to shut him up in a room of Yorimasa RYUZAKI within Koga-jo Castle, and considering Banzan's old age, this punishment might show how impatient the shogunate had become with his criticism.
貞享4年（1687年）幕府は政権批判を行った蕃山が余程憎かったのか69歳と高齢にもかかわらず、松平信之の嫡子である下総国古河藩主松平忠之に預け古河城内の竜崎頼政廓に蟄居謹慎させた。