Victor Riesel was born on the Lower East Side of Manhattan in New York City to Nathan and Sophie Riesel. The family lived in a cold water flat near the elevated railroad tracks. The Riesels were Jewish, and their neighbors were primarily Jewish and Italian American. Victor's father, Nathan, had helped organize the Bonnaz, Singer, and Hand Embroiderers' Union, Local 66, of the International Ladies Garment Workers Union in 1913, and held the Card No. 1 in the local union. In time, Nathan Riesel was appointed a staff member of the union and elected secretary-treasurer and then president of the local union. Victor attended elementary school at (now the Judith K. Weiss School).