Over the next few years, Covay drifted from label to label, eventually signing with Columbia Records in 1961, but success remained elusive. Later that year, however, he had his first chart success, when "Pony Time", a song he co-wrote with fellow Rainbows member John Berry, reached No. 60 on the "Billboard" pop chart. It was issued by the small Arnold label and credited to his group, the Goodtimers. The song was later recorded by Chubby Checker and became a US No. 1 single.