He later moved to New York City, where he played on the album "Dakar" with John Coltrane, played with Lee Morgan on "The Cooker", and briefly worked with Benny Goodman's band in 1958. During this time, Adams also began working with Charles Mingus, performing on one of Mingus's Atlantic albums of the period, "Blues & Roots", which includes Adams' extended solo on "Moanin'". Thereafter, he recorded with Mingus sporadically until the latter's death in 1979. Adams formed a quintet with Donald Byrd in 1958 that lasted until 1961. Following the breakup of the Donald Byrd–Pepper Adams Quintet, Adams lacked a consistent band association until 1965 and the formation of the Thad Jones–Pepper Adams Quintet. During this phase, he performed with the likes of Teddy Charles, Pony Poindexter, Marcus Belgrave, Thelonious Monk, and Lionel Hampton. In September 1963, Adams made an agreement with Motown Records for an exclusive recording contract and an exclusive management contract with International Talent Management, a Motown affiliate. Prior to signing with Motown, Adams turned down an offer from Harry James to play in his Las Vegas-based band because it was extremely commercial and presented few opportunities to solo, despite its $10,000 annual salary.