Moore was born in Durham, North Carolina to Roxie Holloway Moore and Albert P. Moore. Raised in Virginia, he attended college at James Madison University studying tenor saxophone, and later became an accomplished jazz musician in Charlottesville, Virginia, playing with artists such as John D'earth and Dawn Thompson. Moore began playing professionally after a brief stay in college. Moore helped found the Charlottesville Swing Orchestra (1982), and the John D'earth Quintet. The latter played at Miller's, a Charlottesville bar, every Thursday night in the late 1980s, where Moore first met Dave Matthews in 1991. In an effort to bring in instrumental help for some songs Matthews had written, Moore began recording songs with Matthews.