Handler was born in San Francisco, California, the son of Sandra Handler (née Walpole), a retired City College of San Francisco Dean; and Louis Handler, an accountant. His father was a Jewish refugee from Germany, and he is distantly related to British writer Hugh Walpole through his mother. In terms of his early religious upbringing, Handler said, "I had a fairly standard Reform Jewish upbringing, I guess, in terms of the religious side of it." He has a younger sister, Rebecca Handler. Handler has been a voracious reader since childhood, and his favorite author was William Maxwell. He attended Commodore Sloat Elementary, Herbert Hoover Middle School, and Lowell High School. He graduated from Wesleyan University in 1992. He was awarded the 1992 Connecticut Student Poet Prize, which he claims was earned by ripping off Elizabeth Bishop. He is an alumnus of the San Francisco Boys Chorus.