Yarosz lost to the skilled, youthful opponent Billy Conn, another boxer trained by Ray Arcel, on June 30, 1937 in a close twelve round split decision before 13,874 at Forbes Field in Pittsburgh. Yarosz started the first three rounds on offense, scoring heavily with hooks, jabs, and right crosses. Conn looked stronger in the fourth and fifth, scoring with lefts and rights to the head and body. For the sixth through tenth, the well matched duo boxed cautiously, until the final two rounds where both boxers slugged it out, often toe to toe. The excited fans protested the close decision, though Conn was credited with taking the middleweight championship of Pennsylvania. On September 30, 1937, Yarosz lost again to Conn in an equally close fifteen round split decision before 9,000 at Duquesne Gardens in Pittsburgh. Yarosz boxed a smart fight in the first seven rounds, building up a significant points margin. But in the remainder of the long bout, Conn caught up and went ahead on points with a brutal body attack and an occasional right and left to the face that exhausted the older Yarosz who became nearly helpless in the closing rounds. In a fight that had been slow and methodical, the fourteenth opened when Conn dealt a blow to Yarosz's head that made it difficult for him to complete the round, and though he managed, he had little strength left in the fifteenth. Yarosz defeated Conn only once on July 25, 1938, in a twelve-round unanimous decision before a crowd of 10,800 at Forbes Field in Pittsburgh. According to the "Pittsburgh Press" the bout included kidney punching and low left hooks from Conn, and thumbing, gouging and heeling from the gloves of Yarosz. Both fighters, out of anger, continued fighting after the second and fifth rounds. Yarosz outboxed Conn in the last few rounds and Conn was unable to find a remedy or an adequate defense. Yarosz was awarded seven rounds, with just one to Conn, with the seventh even.