Swinburn was born in Oxford. He was the son of Wally Swinburn, who won the Irish flat racing Champion Jockey title in 1976 and 1977 and was the first jockey to record over 100 winners in an Irish flat season. Nicknamed the "Choirboy", he rode his first winner, Paddy's Luck, on 12 July 1978 at Kempton Park but gained considerable fame for riding the superstar Shergar to victory in The Derby in 1981 by a record 10 lengths. Swinburn went on to win the Derby two more times.