He fought six times in 1963, winning four bouts and losing two. He remained ranked in the lower part of the top 10 until December 20, when he surprised the boxing world by flooring past and future world champion Emile Griffith twice in the first round and scoring a technical knockout. That win resulted in "The Ring"s ranking of Carter as the number three contender for Joey Giardello's world middleweight title. Carter won two more fights (one a decision over future heavyweight champion Jimmy Ellis) in 1964, before meeting Giardello in Philadelphia for a 15-round championship match on December 14. Carter landed a few solid rights to the head in the fourth that left Giardello staggering, but was unable to follow them up, and Giardello took control of the fight in the fifth round. The judges awarded Giardello a unanimous decision.