Brought up by poor yet musically talented parents on the island of Funen, he demonstrated his musical abilities at an early age. He initially played in a military band before attending the Royal Danish Academy of Music in Copenhagen from 1884 until December 1886. He premiered his Op. 1, "Suite for Strings", in 1888, at the age of 23. The following year, Nielsen began a 16-year stint as a second violinist in the Royal Danish Orchestra under the conductor Johan Svendsen, during which he played in Giuseppe Verdi's "Falstaff" and "Otello" at their Danish premieres. In 1916, he took a post teaching at the Royal Danish Academy and continued to work there until his death.