Joseph Nathan Oliver was born in Aben, Louisiana, near Donaldsonville in Ascension Parish, and moved to New Orleans in his youth. He first studied the trombone, then changed to cornet. From 1908 to 1917 he played cornet in New Orleans brass bands and dance bands and in the city's red-light district, which came to be known as Storyville. A band he co-led with trombonist Kid Ory was considered one of the best and hottest in New Orleans in the late 1910s. He was popular in New Orleans across economic and racial lines and was in demand for music jobs of all kinds.