Baker began performing as a trombonist in Indianapolis during high school and college. He played in clubs along Indiana Avenue, the heart of the city's jazz scene of the late 1940s and early 1950s, with Jimmy Coe, Slide Hampton, J. J. Johnson, and Wes Montgomery. He mentored Freddie Hubbard and Larry Ridley. He later credited the Hampton family, especially noted jazz trombonist Slide Hampton, for mentoring him in his early years. The Hamptons let him and other local musicians rehearse with their family's jazz band at their Indianapolis home.