Lee lived in the Long Island town of Hewlett Harbor, New York, from 1952 to 1980. They also owned a condominium in Manhattan from 1975 to 1980. He moved in Los Angeles in 1981, they bought a home in West Hollywood, California.