Serrano became a traveling champion, defending his title in places like Venezuela, Ecuador, South Africa, Japan (twice) and, of course, Puerto Rico. One of his defenses, against Julio Diablito Valdez, resulted in an after-fight brawl when Serrano was announced as winner by a unanimous decision and went to greet his rival but was received with a punch to the face. Serrano retaliated, and police intervention was needed. Both fighters were escorted to their dressing rooms by the police. But disaster struck for Serrano in Detroit on August 2 of 1980, when, after leading on all scorecards, he was struck by a Yasutsune Uehara right hand to the chin in round six, and lost his title by knockout to the Japanese. After that loss, he set his eyes on recovering his world title from the Japanese world champion, and so on April 9, 1981, they met again, this time in Wakayama, Japan. Serrano was more cautious this time, and didn't try to go for the knockout after building a points lead. Even though Uehara was fighting in his homeland, all judges agreed and gave Serrano more rounds than they gave Uehara, and Serrano the world title back by a fifteen-round unanimous decision.