So Banzan had to leave the town around Okayama-jo Castle and live a secluded life in Shigeyama Village, Wake County (the present Shigeyama, Bizen City, Okayama Prefecture).
このため、岡山城下を離れ和気郡蕃山村（しげやまそん　現・岡山県備前市蕃山）に隠棲を余儀なくされた。