Lee was born on December 28, 1922, in Manhattan, New York City, in the apartment of his Romanian-born Jewish immigrant parents. His parents were Celia ("née" Solomon) and Jack Lieber, at the corner of West 98th Street and West End Avenue. His father, trained as a dress cutter, worked only sporadically after the Great Depression, His family moved further uptown in Manhattan, in Washington Heights, Manhattan. Lee had one younger brother named Larry Lieber.