Ross was born on November 28, 1937 in Weehawken, New Jersey. He grew up in suburban New Jersey. His father, Wilbur Louis Ross, Sr., was a lawyer who later became a judge, and his mother, Agnes (O'Neill), was a school teacher.