Sime was a member of Duke's baseball and track and field teams, and played football for a season in 1958, while a first-year medical school student. His beginnings in track were accidental: his 100-yard dash on an unmowed grass surface in baseball shoes was a rapid 9.8 seconds, and the coaches soon asked him to join the track team. Opting not to play freshman football, he had gone out for fall track to stay in shape for baseball. Sime hit over .400 as a freshman and had the intention continuing in baseball for coach Ace Parker, but his success during winter track changed that. Parker was a former multi-sport athlete and recognized the exceptional speed and Olympic potential; Sime focused on track in 1956, then split time between both sports in 1957.