he had bought himself homes all over the globe , but rarely left america , dividing his time between his mansion in palm beach , his apartment on fifth avenue and his idyllic beachfront estate on nantucket island .