Having spent his lunch hours studying by himself in Birmingham Central Library, he won a scholarship at the age of 16 to the University of Birmingham. Because his parents believed that a musical career was immoral, they made him leave home and he spent his university time as a fire watcher, living in the fire station, while he studied English literature and music. He began playing the horn. A scholarship to the Royal College of Music to study composition (with Ralph Vaughan Williams and R. O. Morris) and horn (with Frank Probin) followed. Only weeks after starting the course, Probin sent Downes as his deputy on a tour with the London Symphony Orchestra, which continued over the years Downes spent at the College, but on leaving the Royal College he decided that orchestral playing would not be his career. He played in the orchestra at Sadler's Wells in the opening performances of Benjamin Britten's "Peter Grimes" in 1945, and at Covent Garden in ballet performances ("The Sleeping Beauty") in 1946, while still at the Royal College of Music. He also played for the orchestra of the San Carlo Opera Company.