He owned a vacation house in Yanagihara Village, Sunto County, Shizuoka Prefecture (now Numazu City) and an island located off the coast where the house was built is commonly called SAIGO island.
静岡県駿東郡楊原村（現沼津市）に別荘を所有していた（因みに、別荘の沖合に存する島を通称西郷島という）。