The Browns reached the NFL championship game in 1951 and 1952, but lost both times. Speedie led the NFL in receiving in 1951 and was named a first-team All Pro, but he did not play in the championship game due to an injury. He was selected for the Pro Bowl for a second time in 1952. After that season, however, he left the Browns to join the Saskatchewan Roughriders of the Western Interprovincial Football Union under acrimonious circumstances. Speedie had an independent streak that did not sit well with Brown, whose domineering coaching style grated against many of the men who played for him. "He was one of the ones that Paul Brown picked on quite a bit", recalled former teammate Ken Carpenter. "He'd get on Speedie's case for no particular reason." Speedie showed his displeasure by bringing a skunk to training camp in 1952 and calling it "Paul". Brown told Speedie he did not think it was very funny, to which Speedie responded that it was a nocturnal animal and was named after Paul Revere.