As champion, Caldwell won two further bouts before defeating Halimi on points in a rematch at Wembley. In February 1962, a unification bout for the bantamweight title of the world was arranged for São Paulo in Brazil, where Caldwell was to face the legendary Eder Jofre. The Brazilian gradually got on top to stop Caldwell in the tenth. Caldwell, who had been accompanied by his father on the trip, spoke of his memories. ""Eder Jofre was the greatest bantamweight and the hardest-hitter for his weight of all time ... remember the place was packed to the rafters and there were many thousands locked outside the arena. As it turned out, it was my first defeat as a professional and it was hard to take."" While Caldwell sought to regain his title, a chance to guarantee a rematch with Jofre turned up rather closer to home.