In the early days of his retirement, he used to keep more than 10 mistresses in the villa (Ayase sodo (Ayase hermitage)) in Hashiba (Taito Ward, Tokyo), which was known for lots of vacation homes, and enjoyed a gorgeous life of drinking, hanging out with women and creating poetry.
隠居生活は当時、別荘地として知られた橋場（東京都台東区）の別邸（綾瀬草堂）で、妾を十数人も囲い、酒と女と作詩に明け暮れる豪奢な晩年を送った。