Lipsyte was born on January 16, 1938, in New York City, the son of Fanny (Finston) and Sidney I. Lipsyte. He grew up in Rego Park, a neighborhood in the New York city borough of Queens. Lipsyte's father was a school principal, his mother a teacher. Young Robert devoted his childhood to books rather than sports. Instead of sharing a game of catch with his father, the two often visited the library. Robert's son, Sam Lipsyte, is also an author and teacher at Columbia University in New York.