Conn was born in Prestonpans, East Lothian. He joined Hearts from Inveresk Athletic in 1944, making his debut later that year in a 4–0 win over Dumbarton in a wartime Southern League match. He established himself in the first team in the 1948–49 season, when first combined with Bauld and Wardhaugh. Conn's energetic, tenacious style and powerful shooting complemented the cerebral play of Bauld and the dribbling skills of Wardhaugh. Their first match as a forward trio resulted in a 6–1 victory over East Fife. The "Terrible Trio" continued to score freely in the following 5 seasons, Conn notching 102 goals. Hearts became regular top four finishers yet defensive inconsistency ensured they failed to collect any silverware.