Schrager grew up in a Jewish family in Brooklyn. His father Louis owned a factory in Long Branch, New Jersey, which manufactured women's coats and died when Schrager was 19. His mother, Blanche, died when he was 23. In 1968, he graduated from Syracuse University with a BA and then earned a JD from St. John's University School of Law in 1971. While at Syracuse, he was a member and eventual president of the Sigma Alpha Mu fraternity. It was through this fraternity that he met fellow brother Steve Rubell, with whom he would eventually go into business.