After working as assistant conductor of the Los Angeles Philharmonic under Zubin Mehta, Simmons became musical director of the Oakland Symphony Orchestra at age 28; he led the orchestra for four years. He continued to conduct the Los Angeles Philharmonic, both at the Dorothy Chandler Pavilion and at the Hollywood Bowl. He would support Carmen McRae singing jazz one night, then conduct William Walton or Holst's "The Planets" a night or two later. He was the first African-American to be named conductor of a major U.S. symphony orchestra, and was a frequent guest conductor with some of the nation's major opera companies and orchestras (such as the Philadelphia Orchestra). He was the Music Director at the Ojai Music Festival in 1978.