Hoffman was born in Los Angeles, California. He is the younger son of Lillian, a jazz pianist, and Harry Hoffman, who worked as a prop supervisor/set decorator at Columbia Pictures before becoming a furniture salesman. His brother, Ronald, is a lawyer and economist. Hoffman's family was Jewish, although he was not brought up religious. He graduated from Los Angeles High School in 1955.