Torres kept his knockout streak alive through 1962 with three more knockout wins but, in 1963, he suffered his first loss, being stopped in five by Cuba's Florentino Fernández, the only boxer ever to beat Torres by a knockout as a professional. After that setback, Torres went back to training and had one more fight that year, and that time around, he was able to beat another top contender in Don Fullmer, Gene Fullmer's brother, with a ten-round decision win in New Jersey.