Oliver left home at 17 to play with Zack Whyte and his Chocolate Beau Brummels and later with Alphonse Trent. He sang and played trumpet with these bands, becoming known for his "growling" horn playing.