Patterson's amateur record was 44-4 with 37 knockouts. He began fighting as a middleweight and moved to the light heavyweight division. Patterson's first loss was to Joey Maxim in 1954. In January 1956, Patterson married Sandra Hicks. She was his wife until 1969. Their marriage ended because Patterson did not want to quit boxing. 