Schwimmer was born in Flushing, Queens, New York, to Arthur Schwimmer and Arlene Colman. He then lived in Valley Stream, Long Island, until he was two years old. He was raised in Los Angeles, California, where he attended Beverly Hills High School.