In 1936 Gray joined Artie Shaw, who was calling himself Art Shaw, and his "New Music" orchestra as lead violinist. He studied musical arrangement under Shaw and became a staff arranger a year later. During the next two years he penned some of the band's most popular arrangements, including "Carioca", "Softly, As in a Morning Sunrise", "Any Old Time", and the classic "Begin the Beguine." Many of his up-tempo arrangements show early evidence of the style that would eventually become his trademark: a melody broken into two- to four-measure phrases, usually carried by brass section, repeated with increasing intensity until the climax.