but then , he was pretty much isolated from a lot of things living like a recluse himself in his garden district mansion , serving on a rather elitist board of one of the most prestigious museums in the state .