In 1962, he was asked by jazz impresario Norman Granz to become Ella Fitzgerald's full-time accompanist. Flanagan was looking for steadier work than was available with Hawkins, so he accepted. He worked with Fitzgerald from 1962 to 1965. They toured internationally, including to Japan in 1964. He also played with other bands when not required by the vocalist; these included a brief reunion with Rollins in 1965. Later the same year, Flanagan left Fitzgerald and was part of Art Farmer's short-lived New York Jazz Sextet, which recorded "". Flanagan then became accompanist to Tony Bennett for part of 1966, and lived on the West Coast.