Dreyfuss was born on October 29, 1947 in Brooklyn, New York to Norman Dreyfuss, a lawyer, and to Geraldine Dreyfuss, a peace activist. Dreyfuss was raised Bayside, Queens, New York. Dreyfuss and his family first moved to Europe, because his father dislike New York City. Later they moved to Los Angeles, California when he was just 9 years old. He was educated at Beverly Hills High School.