Zimmer was born on August 16, 1944 in Newark, New Jersey to William and Evelyn Zimmer, the second of two children. In his early years he was raised in Hillside, New Jersey. His father, a physician, died of a heart attack when he was 3 years old. After his father's death, his mother moved from Hillside to Bloomfield, New Jersey, where she supported the family by working as a clerk at the Sunshine Biscuits warehouse. They lived in a Bloomfield garden apartment, which Zimmer has referred to as "the New Jersey equivalent of a log cabin."